神经网络结构复杂度及其功能表现：梯径理论与算法信息论

npjComplexity

|(2024)1:151

npj|complexityArticle

/10.1038/s44260-024-00015-x

Correlatingmeasuresofhierarchical

structuresinartificialneuralnetworkswiththeirperformance

Checkforupdates

ZhuoyingXu1

,6

,YingjunZhu1

,6,BinbinHong

2

,XinlinWu3

,4

,5

,JingwenZhang3

,4

,5

,MufengCai3

,4

,5

,DaZhou

1

区&YuLiu

3

,4

区

ThisstudyemploystherecentlydevelopedLadderpathapproach,withinthebroadercategoryof

AlgorithmicInformationTheory(AIT),whichcharacterizesthehierarchicalandnestedrelationshipsamongrepeatingsubstructures,toexplorethestructure-functionrelationshipinneuralnetworks,

multilayerperceptrons(MLP),inparticular.Themetricorder-rateη,derivedfromtheapproach,isameasureofstructuralorderliness:whenηisinthemiddlerange(around0.5),thestructureexhibitstherichesthierarchicalrelationships,correspondingtothehighestcomplexity.Wehypothesizethatthehigheststructuralcomplexitycorrelateswithoptimalfunctionality.Ourexperimentssupportthis

hypothesisinseveralways:networkswithηvaluesinthemiddlerangeshowsuperiorperformance,andthetrainingprocessestendtonaturallyadjustηtowardsthisrange;additionally,startingneuralnetworkswithηvaluesinthismiddlerangeappearstoboostperformance.Intriguingly,thesefindings

alignwithobservationsinotherdistinctsystems,includingchemicalmoleculesandproteinsequences,hintingatahiddenregularityencapsulatedbythistheoreticalframework.

Itiswidelyrecognizedthattheperformanceofneuralnetworksdependsontheirarchitecture

1

.Ononehand,thedesignofthesestructuresisinspiredbybiologicalneuralnetworks.Forinstance,convolutionalnetworksdrawfromtheconceptofvisualreceptivefields

2

,3

.Ontheotherhand,theintroductionofnewnetworkarchitecturesoftenleadstosignificantperformanceleaps,suchastheTransformer,whichincorporatesaspectsofattentionmechanisms

4

.Whileitisknownthatthearchitectureofartificialneuralnetworksisakeydeterminantoftheirfunctionalityandperformance,inpractice,thedevelopmentofneuralnetworksismovingtowardsincreas-inglylargerscales,morecomplexstructures,andgreaternumbersoflayers.Forexample,thescaleofnetworks,intermsofthenumberofparameters,hasgrownfromthethousandsinLeNettothemillionsinAlexNet,andthentothebillionsinGPT;whilethearchitecturehasevolvedfromLeNettoAlexNet,VGG,GoogLeNet,variousAutoencoders,andmorerecentlytoTransformer

5

-9

.

Whilelarge-scalenetworksdemonstrateformidablecapabilitiesinpracticalapplications,theincreasedcomplexityofneuralnetworksimpliesaneedformoretrainingdata,longertrainingtimes,higherexpenses,andagrowingconcernforenergyconsumption

10

.Ifitwerepossibletoestimatetheappropriateneuralnetworkstructureandits

complexityforspecifictasksinadvance,substantialcomputationalresourcescouldbesaved.Additionally,insituationswherecompu-tationalresourcesarelimited,suchasinsmalldevicesorspaceequipment,itmightbenecessarytousethesmallestpossibleneuralnetworks

11

.Therefore,anunderstandingofthegenerallawsgov-erningtherelationshipbetweenthestructureandperformanceofneuralnetworksalsoholdssignificantpracticalvalue.

Researchersareactivelyinvestigatingtherelationshipbetweenneuralnetworkstructuresandtheirperformancefromvariousperspectives.Byrepresentingneuralnetworksasgraphs,onecanemployexistinggraphmetricsordefinenewonestoelucidatethelinkbetweennetworkperfor-manceandthesemetricsacrossdifferentapplications

12

,13

.Someresearchdirectlyfocusesontheparticularstructureofnetworks

14

-16

,whileotherstudiesintroducenovelneuralnetworkstructurestoexaminetheirimpactonnetworkperformanceenhancement

17

.Thesestudieshaveraisednumerousfundamentalquestionsthatareurgentlyinneedofbeinganswered.Questionssuchaswhetherthereisasystematicrelationshipbetweenneuralnetworkstructuresandtheirperformance,orwhetherhigh-performingneuralnetworkspossessdistinctstructuralcharacteristics,arecentraltothisresearcharea.

1SchoolofMathematicalSciences,XiamenUniversity,Xiamen,China.2DepartmentofPhysics,FacultyofArtsandSciences,BeijingNormalUniversity,

Zhuhai,China.3DepartmentofSystemsScience,FacultyofArtsandSciences,BeijingNormalUniversity,Zhuhai,China.4InternationalAcademicCenterof

ComplexSystems,BeijingNormalUniversity,Zhuhai,China.5SchoolofSystemsScience,BeijingNormalUniversity,Beijing,China.6Theseauthorscontributedequally:ZhuoyingXu,YingjunZhu.e-mail:

zhouda@

;

yu.ernest.liu@

/10.1038/s44260-024-00015-x

Article

Hierarchyisanimportantfeatureamongvariousstructuralchar-acteristics.Intheresearchintobothartificialandbiologicalneuralnetworks,theimportanceandthepotentialuniversalityofhierarchicalstructureshavebeendiscovered(seethetwoparagraphsattheendofthissection).However,quantitativelycharacterizingthehierarchicalstructureofneuralnetworksisnotatrivialtask.TheLadderpathapproach,arecentlydevelopedmathe-maticaltoolwithinthebroadercategoryofAlgorithmicInformationTheory(AIT),offersarigorousanalyticalapproachforexaminingthestructuralinformationofatargetsystem

18

,19

.

Thisapproachpinpointsrepeatingsubstructures(or“modules”)withinthetargetsystem,whichthemselvesmayconsistofevensmallerrepeatingunits,progressivelybreakingdowntothemostbasicbuildingblocks.Theserepeatingsubstructurescanthenbereorganizedintoahier-archical,modular-nested,tree-likestructure.Oneextremescenarioisasystemwithaflattenedhierarchy,essentiallydevoidofrepeatingstructures,andcanbelikenedtocompletelydisorderedstructuresorentirelyrandomgeneticsequences.Theotherextremeinvolvessimplerepetitionofsmallsubstructures,likeadoublingsequence(2becomes4,4becomes8,8becomes16,andsoforth),whichiscomparabletocrystals.Systemsdeviatingfromtheseextremesandpositionedinthemiddlearethosewitharichhierarchicalstructure.Wehypothesizethatthedegreeofrichnessinthishierarchicalstructureispositivelycorrelatedwiththeperformanceofneuralnetworks.

TheLadderpathapproachhasrecentlybeensuccessfullyappliedinlivingandchemicalsystems,suchasanalyzingtheevolutionofproteinsequences

19

;asimilarmethodhasalsobeenusedtoinvestigatetheoriginsoflifethroughtheanalysisofmolecularstructure

20

.Theapplicabilityofthisapproachintheseareasislinkedtothetightstructure-functionrelationshipprevalentinevolutionarysystems(e.g.,thestructureofmoleculesdictatestheirphysicochemicalproperties,andthestructureofdrugmoleculesdeterminestheirbindingwithreceptorproteins).Similarly,thisstructure-functionrelationshipislikelytoexistinanotherkeyevolutionarysystem:intelligence.ThispaperemploystheLadderpathapproachtoanalyzethestructure-functionrelationshipinartificialneuralnetworks.Weacknowl-edgethatourworkiscurrentlylimitedtomultilayerperceptrons(MLP)andnetworksofconstrainedsizesduetothetechnicalchallengesofladderpathcalculations,aswehaveshownthatthesecalculationsareanNP-hardproblem

18

.However,ourprimarymotivationistodemonstratetheeffec-tivenessandfeasibilityofthisAITapproachinstudyingthehierarchicalandnestedstructuresofartificialneuralnetworks.Moreimportantly,coupledwiththerecentworkonproteinsequencesasdemonstratedin

19

,thistypeof

structure-functionrelationshipappearstomanifestinseveralseemingly

unrelatedsystems,suggestingacertainuniversalitythatmeritsattention.

Beforemovingforward,letusreviewthepreviousliteratureonhier-archicalstructures.Thefirstpartisabouthierarchicalstructuresinartificialneuralnetworks.Forcertainspecificnetworks,suchasreservoircomputing,whichisanincreasinglyprominentneuralnetworkframeworkforstudyingdynamicalsystems

21

,researchhasshownthattheircomputationalcapacityismaximizedwhentheyareinacriticalstate

22

(althoughsomescholarsbelievethatthismaximizationofcomputationalcapacityisconditionalandonlytrueunderspecificcircumstances

23

).In2021,Wangetal.discoveredthatinwell-trainedreservoirnetworks,thenodessynchronizeinclusters,andthesizedistributionofthesynchronizationclustersfollowsapowerlaw

15

.Itisimportanttonotethatapower-lawdistributionisoftenakeycharacteristicofacriticalstate,andthisdistributionistypicallyamanifes-tationofarichlyhierarchicalstructure

24

.In2020,Leskovecetal.proposedamethodtorepresentneuralnetworksasgraphs,termedrelationalgraphs,whichcharacterizetheinter-layerconnectionsofneuralnetworks

13

.Theydiscoveredthatincaseswhereaneuralnetworkperformswell,theaveragepathlengthandclusteringcoefficientofitscorrespondingrelationalgraphtendtoalwaysfallwithinacertainrange(itisnotablethatthesetwoindicesintuitivelyrepresentcharacteristicsthataretypicallyoppositeinnature).Furthermore,thetrainingprocesstendstoshiftthesetwostructuralmetricstowardsthisrange.In2018,Yingetal.introducedadifferentiablegraphpoolingmethod,designedtointegratecomplexhierarchicalrepresentations

ofgraphs,andfurtherconnectdeepergraphneuralnetworkmodelstotheserepresentations

16

.Thismethodhassignificantlyenhancedaccuracyingraphclassificationbenchmarktaskscomparedtootherpoolingmethods,underscoringthecriticalroleofutilizinghierarchicalinformationforneuralnetworkperformance.In2016,Bengioetal.proposedthreecomplexitymetricsforthearchitectureofrecurrentneuralnetworksfromagraphperspective:recurrentdepth,feedforwarddepth,andrecurrentskipcoeffi-cient.Experimentshaverevealedthatincreasingrecurrentandfeedforwarddepthcanenhancenetworkperformance,andaugmentingtherecurrentskipcoefficientcanimproveperformanceintaskswithlong-termdependencies

12

.

Then,thisparagraphcovershierarchicalstructuresinbiologicalneuralnetworks.Inrealbiologicalneuralnetworks,similarstudieshavealsoshownapositivecorrelationbetweenhierarchicalstructureandgoodperformance.Baumetal.analyzedsamplesfromyouthsaged8–22inthePhiladelphiaNeurodevelopmentalCohorttostudytheevolutionofstructuralbrainnetworksovertime

25

.Theyobservedthatasindividualsage,thenetworkstructureundergoesamodularseparationprocess,whereconnectionswithinmodulesstrengthenandinter-moduleconnectionsweaken.Fur-thermore,thisseparationappearstobebeneficialforthedevelopmentofbrainfunctionsinyouths.Vidaurreetal.,usingwhole-brainresting-statefMRIdata,employedmethodssuchashiddenMarkovchainsandhier-archicalclusteringtoanalyzetheorganizationaldynamicsofbrainnetworksovertime

26

.Theydiscoveredadistincthierarchicalstructurewithinbraindynamics,whichshowssignificantcorrelationswithindividualbehaviorandgeneticpredispositions.Theirresearchunderscoresacloseconnectionbetweenthehierarchicalstructureandfunctioninactualbrainnetworks.In2021,Zhouetal.studiedtherelationshipbetweennetworkstructureanditsdynamicpropertiesfromdifferentperspectives.Theydiscoveredthatmodularnetworktopologiescansignificantlyreducebothoperationalandconnectivitycosts,thusachievingajointoptimizationofefficiency

27

.Alsoin2021,Luoreviewedcommoncircuitmotifsandarchitecturalplans,exploringhowthesecircuitarchitecturesassembletoachievevariousfunctions

28

.Thesecircuitmotifscanbeconsideredas“words”,whichcombineintocircuitarchitecturesthatmightoperateatthelevelof“sen-tences”.Thisworkemphasizestheimportanceofhierarchicalstructures:onlyaftergainingabetterunderstandingofthenestedmodularhierarchicalstructuresamongtheseneuronalconnectionscanwebegintounderstandthe“paragraphs”(e.g.,brainregions)andeventuallythe“article”,whichmayinspiremoreimportantadvancesinartificialintelligence.

Methods

RecaptheLadderpathapproach

TheLadderpathapproach,detailedin

18

,19

,offersaquantitativeapproachtoanalyzestructuralinformationinsystems,rangingfromsequences,mole-culestoimages.Ititerativelyidentifiesrepeatingsubstructures,calledlad-derons,whichareessentiallyreusedmodules.Thesesubstructuresmaybecomposedofsmallerrepeatingunits,cascadingdowntothesystem’smostbasicbuildingblocks.Theseladderonscollectivelyformahierarchical,nested,tree-likestructureknownasaladdergraph.

UsingthesequenceABCDBCDBCDCDACAC(denotedasX)asanexampletoillustrate,theladdergraphiscomputedasshowninFig.

1

,demonstratinghowtoconstructthetargetsequenceXfromthefourbasicbuildingblocksA,B,C,andDinthemostefficientmanner.First,combineCandDtoconstructCD,andAandCtoconstructAC,whichtakes2steps;thencombiningBwithCDtoconstructBCDisanotherstep.Next,con-structthetargetsequenceXusingalreadygeneratedsubstructures(or“modules”)andbasicbuildingblocks.ThisinvolvescombiningA,BCD,BCD,BCD,CD,AC,andAC,thustaking6steps;intotal,thisis9stepssofar;thefinalstepinvolvesoutputtingX.Therefore,constructingXinthemostefficientmannertakes10stepsintotal.

Xhas16letters,butonly10stepswereneededduetothereuseofsomerepeatingsubstructures:forexample,thepreviouslyconstructedCDisreusedinconstructingBCD;whileBCDappearsthreetimes,thesubsequentinstancesofBCDcandirectlyusetheinitiallyconstructedBCD,saving

npjComplexity

|(2024)1:152

/10.1038/s44260-024-00015-x

Article

Fig.1|Laddergraphsofdifferentsystems.aLaddergraphofashortsequence,forillustrativepurposes.Thegrayhexagonsrepresentthebasicbuildingblocks,andthegraysquaresrepresenttheladderons.Inb–d,basicbuildingblocksareomitted,andladderonsarerepresentedasgrayellipses.bLaddergraphofacompletelyrandomanddisorderedsequence,withη=0.04,BBDCDDCAACABACCDADA-

BABDDD...,whichshowsminimalhierarchicalrelationshipsamongrepeating

substructures.cLaddergraphofasequencecomposedentirelyofA's(themost

orderedsequence),withη=1.TheladderonsareAA,AAAA,AAAAAAAA,etc.dLaddergraphofthesequenceABABABDABABBAABAABCACABABDA...,

whichhasη=0.53.Thissequencedisplaystherichesthierarchicalstructure.Thesequencesshownin(b-d)areeachcomposedofthebasicbuildingblocksA,B,C,andD,andhavealengthof300,representingthreetypicalcategoriesofsequences.

manysteps.ThisprincipleofreuseisafundamentalconceptinAITandtheLadderpathapproach.

Notethatinourexample,somestepscanbeinterchanged,suchaswhethertoconstructCDorACfirst,whichdoesnotmatter.However,theorderbetweenCDandBCDmustnotbereversed,becauseBCDisbasedonCD.Therefore,theladdergraphalsocorrespondstoapartiallyorderedmultiset,whichcanbedenotedas

{B;D;A(2);C(2)/AC;CD/BCD(2)}(1)

Stepswithinthesamelevel(i.e.,separatedby“/”)canbeinterchanged,butnotacrosslevels.Thisiswhythesequenceexhibitsahierarchicalstructure.

Now,wecanintroduceseveralimportantnotionswithintheLadder-pathapproach.Intheexampleabove,the“10steps”aredefinedastheladderpath-indexofX,whileanotherquantity,theorder-indexω,isdefinedasthelengthofXminusitsladderpath-index,whichisω(X)=16-10=6.Mathematically,wehaveshowninref.

18

thatωalwaysequalsthesumofthe“reducedlengths”lofeachladderon(wherethereducedlengthisdefinedasthelengthofeachladderonminusone).Inthiscase,ω(X)=lAC+lCD+2×lBCDwherelACisthelengthofACminus1,namely(2−1),lCD=2−1,andlBCD=3−1;Themultiplierof2forlBCDisbecauseitsmultiplicityinthepartiallyorderedmultiset,asshowninEq.(

1

),is2,indicatingthatitwasreusedtwice.Thus,theorder-indexωcountsthesizesofallrepeatingsubstructuresinasystem,therebyessentiallychar-acterizinghoworderedasystemisinanabsolutemeasure.Foramoredetailedtheoreticalexplanationandmathematicalderivation,pleaserefertoref.

18

.

Thisrecapmaybeconsideredlengthy,whichcouldseemover-whelming,andsomenotionsmightappearunnecessaryatfirstglance.Infact,thisisbecausetheLadderpathapproachwithinAITwasnotspecificallydevelopedtocharacterizeneuralnetworksbutwasdevelopedfromamoregeneralperspective,whichmaymakeitseemverbose.However,ithasbeensuccessfullyappliedinseeminglyunrelatedfieldssuchasproteinsequences

19

,andoursubsequentfindingsalsodemonstratethatthisapproachperformswellinartificialneuralnetworks.Thisindicatesthattheapproachandtherelationshipsbetweenthehierarchicalstructureand

performanceitdescribesareuniversaltoacertainextent,makingitworthwhile.

Ladderpathcharacterizeshierarchicalstructures

Aftertherecap,weproceedbytakingmorecomplexsequencesasexamples.Wewilldemonstratetheladdergraphsofthreetypicalcategoriesofsequences:minimalrepetition,akintocompletelyrandom,disorderedsystems(Fig.

1

b);simplerepetition,similartoacrystallinestructure,wherethepatternprogressesfrom2to4,4to8,8to16,andsoon(Fig.

1

c);andtheonesthatliebetweenthesetwoextremes,showcasingtherichesthierarchicalstructure(Fig.

1

d).

Tobettercharacterizethehierarchicalstructure,wehave,inref.

19

,definedarelativemeasureontopoftheabsolutemeasureω,calledtheorder-rateη.Tobeginwith,letusfirstexaminethedistributionofωofvarioussequencesversustheirlengthsS,asshowninFig.

2

.ForagivenlengthS,wecanobservethatωhasbothamaximumandaminimumvalue:themax-imum,denotedasωmax(S),correspondstosequencesthatarecompletelyidentical;theminimum,denotedasω0(S),correspondstopurelyrandomsequences.Theminimumvaluearisesbecause,insequenceswithafinitenumberofbases(herebeingA,B,C,andD),repeatingsubstructureswillinevitablyappearasthelengthincreases,meaningeventhepurelyrandomsequencewillnothaveanωofzero.Therefore,weneedtonormalizeit,leadingtothedefinitionoftherelativemeasureorder-rateη(x)forsequencex:

η(x):=ω(x)-ω0(S)(2)

ωmax(S)-ω0(S)

whereSisthelengthofthesequencex.Anηof0meanscompletedisorder(Fig.

1

b),1impliesfullorder(Fig.

1

c),andaround0.5indicatesarichlystructuredhierarchy(Fig.

1

d).

Infact,intheLadderpathapproach,ηisrelatedtothecomplexityofasystem.Asystemisnotconsideredcomplexifentirelyrandom(η≈0)orordered(η≈1);complexityemergesonlyintheintermediatestate(η≈0.5).ThisdistinguishesLadderpathfromsimilarconceptssuchasKolmogorov

npjComplexity

|(2024)1:153

/10.1038/s44260-024-00015-x

Article

npjComplexity

|(2024)1:154

complexity,additionchain,assemblytheory,andthe“adjacentpossible”

29

–31

.

Inthecontextofneuralnetworks,wecanapplytheLadderpathapproachtosystematicallyorganizethenetwork’srepeatingsub-structuresinanestedhierarchicalmanner.AsillustratedinFig.

3

a,b,thepartshighlightedbyred,yellow,andgreenlinesrepresentthesesubstructures(withthesamecolorindicatingidentical,reusedmodules).Fromthis,wecanestablishahierarchicalrelationship(andillustratetheladdergraph):theredsubstructureisencompassedwithinboththeyellowandgreenones.Nevertheless,directlycalcu-latingtheladdergraphisquitechallenging(andthisproblemisinherentlyNP-hard

18

).Hence,wefirsttransformthenetworkintoa

Fig.2|Distributionoftheorder-indexωforsequencesofvaryinglengths,illus-tratingthecalculationoftheorder-rateη.

setofsequencesandthencomputeit(themethodforthistrans-formationwillbedetailedinsection“Sequencerepresentationofaneuralnetwork”,andtherearealreadyalgorithmsdevelopedforcomputingladdergraphsofsequencesatthescaleof10,000inlength

19

).Finally,wehypothesizethattheneuralnetwork’sabilitytoextractandintegrateinformationisatitspeak(achievingitsbestperformance)whenitshierarchicalstructureistherichest(i.e.,whenmodulereuseandtinkeringaremostpronounced),correspondingtotheorder-rateηaround0.5.

Experimentsetup

Toinvestigatetheconnectionbetweenthestructureofaneuralnetworkanditsfunctionality,wechosearathersimpletask:recognizingwhetherathree-digitnumberisoddoreven.ThiswasaddressedusingaMLP.Theinputconsistsofthreeneurons,representingthehundreds,tens,andonesplace,respectively,whiletheoutputhastwoneuronsindicatingoddoreven.Thehiddenlayersvary,either1,2,3,or4layers,eachcomprisingadifferentnumberofneurons.Tosimplifytheanalysis,welimitedeachMLPtoamaximumof200edges.Withthisconstraint,anMLPwithonehiddenlayercouldhaveonly40distinctvariations:Giventheinputlayerhas3neuronsandtheoutputlayerhas2neurons,ifthehiddenlayerhas40neurons(wecandenotethisMLP[3,40,2]),thereareatotalof3×40+40×2=200edges;theMLP[3,41,2]wouldhave205edges.ForanMLPwithtwo,three,andfourhiddenlayers,weconstructed200distinctarchitecturesforeachcategory(suchas[3,7,17,2],[3,8,3,5,2],and[3,3,5,14,2]whichhave2,3,and3hiddenlayersrespectively),contributingtothetotalof640differentarchitectures.

Thenetworkconnectionweightswererandomlyinitialized,andall640architecturesunderwentanidenticaltrainingperiod,eachfor2000epochs.Duringthetraining,wemonitoredthechangesinperformance(measuredbyaccuracy)andthenetwork’sorder-rateη(calculatedbasedontheLad-derpathapproach),toexploretherelationshipbetweentheseaspects.Sec-tion“Results”presentstheevolutionaryandstatisticalrelationshipsbetweenstructureandfunction.

Fig.3|ThediagramillustratinghowtoemploytheLadderpathapproachtoanalyzeaneuralnetwork,andreorganizetherepeatingsubstructuresintoaladdergraph.aAschematicdiagramofanMLP.bAschematicdiagramoftheladdergraphofthisMLP.cRepresentinganMLPasasetofsequences.

/10.1038/s44260-024-00015-x

Article

Sequencerepresentationofaneuralnetwork

ToutilizethealgorithmbasedontheLadderpathapproach

19

forstudyingthestructure-functionrelationshipinneuralnetworks,weneedtouseasequenceorasetofsequencestorepresentaneuralnetwork’sstructure.Byenvisioninganeuralnetworkasasignal(i.e.,theinput)propagationpro-blemoraninformationflowproblemwithinanetwork,thecollectionofallpathsthatthesesignalstraversecanrepresentthenetworkstructure.Sincetheconnectionweightsbetweenneuronsarerealnumbers,wefirstneedtocoarse-grainthem,usingdifferentsymbolstorepresentweightswithinthesamerange.

Thehigherthedegreeofcoarse-graining,themoreconducivetheextractedinformationisforsequenceanalysis;thelowerthedegreeofcoarse-graining,themoreinformationretained.However,thisalsoresultsinmoreunnecessarydetails,whichcanmakesubsequentanalysismoredif-ficult.Therefore,weconductedexperimentswithvariousdegreesofcoarse-grainingtofindabalancethatisaslargeaspossiblewithoutsignificantlydiminishingfunctionalperformance.Fortheparticularsystemsweselected,ournumericalexperimentsshowedthatwhenthecoarse-grainingintervalis

Order-rateη

Fig.4|Thedistributionoftheorder-rateηvs.theaccuracyintheodd-even

recognitiontaskperformedbytheneuralnetworks.Thestatisticsincludealloftheaforementioned640differentarchitectures.

settoamaximumof0.1,ithasaminimalimpactontheneuralnetwork’sperformance(seeSupplementaryNote1fordetailedinformation).

Subsequently,weobtaintheappropriatelycoarse-grainedgraph,andwecanconvertthegraphintoasetofsequences.Neuronsarethenodesofthisgraph,andnodeswithinthesamelayerareassignedthesamesymbolduetotheirsharedactivationfunction.Connectionsbetweenneuronsaretheedgesofthegraph,andedgeswithuniqueweights(discretizedaspre-viouslymentioned)areconsideredtocarrydistinctinformation,andarethusrepresentedusingdifferentsymbols.Forexample,thepathhighlightedingrayinFig.

3

canbedenotedasAzBxC.Throughthismethod,wecandetaileverypathaninputsignaltraverses,andaggregatethesepathsintoasetofsequencestodepictthegraph.WecanthenemploytheLadderpathapproachtoexaminethesequences,andtherebyinvestigatethechar-acteristicsoftheneuralnetwork’sstructure.

Results

Bestperformancewhenthehierarchicalstru