DeepHyd：一种基于深度学习的人工智能方法用于根据激光雷达和声纳数据对水工结构进行自动分类

DeepHyd:ADeepLearning-basedArtificialIntelligenceApproachfortheAutomatedClassificationofHydraulicStructuresfromLiDARandSonarData

WenwuTang,Ph.D.Shen-EnChen,Ph.D.JohnDiemer,Ph.D.CraigAllan,Ph.D.

TianyangChen,GraduateResearchAssistantZacherySlocum,GraduateResearchAssistantTariniShukla,GraduateResearchAssistant

VidyaShubhashChavan,Ph.D.,FormerGraduateResearchAssistantNavanitSriShanmugam,GraduateResearchAssistant

CenterforAppliedGeographicInformationScienceDepartmentofGeographyandEarthSciencesDepartmentofCivilandEnvironmentalEngineeringSchoolofDataScience

UniversityofNorthCarolinaatCharlotte

NCDOTProject2019-03FHWA/NC/2019-03

i

January2022

PAGE\*roman

viii

DeepHyd:ADeepLearning-basedArtificialIntelligenceApproachfortheAutomatedClassificationofHydraulicStructuresfromLiDARandSonarData

FinalReport

(ReportNo.FHWA/NC/2019-03)

To:NorthCarolinaDepartmentofTransportation(ResearchProjectNo.RP2019-03)

Submittedby

WenwuTang1,2,3,*,Shen-EnChen4,*,JohnDiemer2,*,CraigAllan1,2,*,TianyangChen1,2,**,ZacherySlocum1,2,**,TariniShukla1,4,**,VidyaShubhashChavan4,**,NavanitSriShanmugam4,**

1CenterforAppliedGeographicInformationScience

2DepartmentofGeographyandEarthSciences

3SchoolofDataScience

4DepartmentofCivilandEnvironmentalEngineeringUniversityofNorthCarolinaatCharlotte,Charlotte,NC28223

*:PIs

**:GraduateResearchAssistants

ContactWenwuTang,Ph.D.

Phone:(704)687-5988;

Fax:(704)687-5966;

Email:

wtang4@

January2022

TechnicalReportDocumentationPage

1. ReportNo.

FHWA/NC/2019-03

2. GovernmentAccessionNo.

…leaveblank…

3. Recipient’sCatalogNo.

…leaveblank…

4.TitleandSubtitle

DeepHyd:ADeepLearning-basedArtificialIntelligenceApproachfortheAutomatedClassificationofHydraulicStructuresfromLiDARandSonarData

5. ReportDate

January30th,2022

6. PerformingOrganizationCode

…leaveblank…

7. Author(s)

WenwuTang,Shen-EnChen,JohnDiemer,CraigAllan,TianyangChen,ZacherySlocum,TariniShukla,VidyaShubhashChavan,NavanitSriShanmugam

8. PerformingOrganizationReportNo.

…leaveblank…

9. PerformingOrganizationNameandAddress

CenterforAppliedGeographicInformationScienceDepartmentofGeographyandEarthSciences,

UniversityofNorthCarolinaatCharlotte,Charlotte,NC28223

10.WorkUnitNo.(TRAIS)

…leaveblank…

11.ContractorGrantNo.

…leaveblank…

12.SponsoringAgencyNameandAddressNorthCarolinaDepartmentofTransportationResearchandDevelopmentUnit

13.TypeofReportandPeriodCoveredFinalReport

104FayettevilleStreetRaleigh,NorthCarolina27601

07/01/2018-12/31/2021

14.SponsoringAgencyCodePR2019-03

SupplementaryNotes:

…leaveblank…

16.Abstract

Monitoringtheconditionsofhydraulicstructuressuchasbridgesandculvertsisessentialinwarrantingthesafetyandsustainabilityoftransportationinfrastructure.ThisisparticularlyimportantforNorthCarolinaasmorethan8percentofNCbridgeshavebeenfoundinpoorconditionsandneedimmediatemaintenance.Lidarandsonartechnologieshavebeenincreasinglyappliedtosupportthismonitoringneed.However,theprocessingandclassificationofpointclouddatageneratedfromLiDARandsonartechniquesrepresentsachallengeashydraulicstructuresareoftencomplicatedingeometriccharacteristicsandconsiderablelaborandtimeareneededfortheprocessingandclassification.

Toaddressthischallenge,inthisproject,wedevelopedDeepHyd,adeeplearning-based3Dmodelingframeworkandsoftwaretoolsfortheautomatedclassificationofpointclouddataofhydraulicstructures.Wecollectedfielddatafrom11sitesintheGreaterCharlotteMetropolitanregionfortrainingandvalidationofthedeeplearningalgorithms.ThefielddatacollectioncombinestheuseofterrestrialLiDAR,sonar,totalstation,survey-gradeGPS,anddrone.Thedeeplearningalgorithmthatweusedforpointcloudclassificationisastate-of-the-art3Dartificialintelligencetechnique.Weusedatwo-tieredmodelingapproachtotraindeeplearningalgorithmsusingannotatedpointclouddata:classificationofbridgesfromvegetationandground,andclassificationofspecificbridgecomponentsincludingbeam,pier,railing,andretainingwalls.Weimplementedscientificworkflowstoautomatetheclassificationofpointclouddataofhydraulicstructuresusingdeeplearning.Ourmajorfindingsare:

1)our3DdeeplearningalgorithmsinDeepHydachievehighclassificationperformanceonpointclouddataofhydraulicstructures.2)deeplearningcaneffectivelyhandletheclassificationoflargevolumesofpointclouddata,butthetrainingofdeeplearningalgorithmsrequireslargeamountsofannotateddata.3)annotatedpointclouddataserveasafoundationdatabasefortheautomatedclassificationofhydraulicstructuresusingartificialintelligencetechniques.Moreannotatedpointclouddata,whichcoveralternativetypesofhydraulicstructures,areneededforfurtherimprovingclassificationperformance.

17.KeyWords

Pointcloudclassification,Hydraulicstructures,Deeplearning,ArtificialIntelligence

18.DistributionStatement

…leaveblank…

19.SecurityClassif.(ofthisreport)Unclassified

20.SecurityClassif.(ofthispage)Unclassified

21.No.ofPages60

22.Price

…leaveblank…

FormDOTF1700.7(8-72) Reproductionofcompletedpageauthorized

Disclaimer

Thecontentsofthisreportreflecttheviewsoftheauthor(s)andnotnecessarilytheviewsoftheUniversity.Theauthor(s)areresponsibleforthefactsandtheaccuracyofthedatapresentedherein.ThecontentsdonotnecessarilyreflecttheofficialviewsorpoliciesofeithertheNorthCarolinaDepartmentofTransportationortheFederalHighwayAdministrationatthetimeofpublication.Thisreportdoesnotconstituteastandard,specification,orregulation.

Acknowledgements

ThisprojectissupportedbytheNorthCarolinaDepartmentofTransportation.Specifically,theauthorsowethankstotheSteeringandImplementationCommittee:includingMatthewLauffer(Chair),JohnW.Kirby,TomLangan,GaryThompson,PaulJordan,MarkSwartz,MarkWard,DerekBradner,BrianRadakovic,KevinFischer.WethankMatthewMacon,RodneyHough,DonaldEarly,fromUASprogramandPhotogrammetryUnit,NCDOTfortheirhelponUAStesting.WethankNCDOTITfortheirhelpondeployingandtestingthesoftware.

ExecutiveSummary

Monitoringtheconditionsofhydraulicstructuressuchasbridgesandculvertsisessentialinwarrantingthesafetyandsustainabilityoftransportationinfrastructure.ThisisparticularlyimportantforNorthCarolinaasmorethan8percentofNCbridgeshavebeenfoundinpoorconditionsandneedimmediatemaintenance.LiDARandsonartechnologieshavebeenincreasinglyappliedtosupportthismonitoringneed.However,theprocessingandclassificationofpointclouddatageneratedfromLiDARandsonartechniquesrepresentsachallengeashydraulicstructuresareoftencomplicatedintheirgeometriccharacteristicsandconsiderablelaborandtimeareneededfortheprocessingandclassificationoflargepointclouddatasets.

Toaddressthischallenge,inthisproject,wedevelopedDeepHyd,adeeplearning-based3Dmodelingframeworkandsoftwaretoolsfortheautomatedclassificationofpointclouddataofhydraulicstructures.Wecollectedfielddatafrom11sitesintheGreaterCharlotteMetropolitanregionforthetrainingandvalidationofthedeeplearningalgorithms.Thefielddatacollectioncombinestheuseofavarietyofsurveyinstruments,includingterrestrialLiDAR,sonar,totalstation,survey-gradeGPS,anddrone-basedphotogrammetry.Thedeeplearningalgorithmthatweutilizedforthepointcloudclassificationisastate-of-the-art3Dartificialintelligencetechniquebasedonconvolutionalneuralnetworks.Weusedatwo-tieredmodelingapproachtotraindeeplearningalgorithmsusingannotatedpointclouddata:classificationofbridgesfromvegetationandground,andclassificationofspecificbridgecomponentsincludingbeam,pier,railing,andretainingwalls.Weimplementedscientificworkflowstoautomatetheprocessingandclassificationofpointclouddataofhydraulicstructuresusingdeeplearning.

ConsideringtheuniquegeographicaldivisionsinNorthCarolinafromthemountainridgesintheAppalachiantotheAtlanticcoastalplain,agreatdiverseofhighwaybridgetypesinterconnectedthestateandtheautomatedbridgecomponentclassificationtoolrepresentsaparadigmshiftintransportationmanagement.Ourmajorfindingsaresummarizedbelow:

Our3DdeeplearningalgorithmsinDeepHydachievehighclassificationperformanceonpointclouddataofhydraulicstructures.Thetwo-tieredgeospatialmodelingdesigncaneffectivelysupport1)theclassificationofhydraulicstructures,vegetation,andgroundsurfaces,and2)theclassificationofspecificbridgecomponents.

Transferlearningusingpre-trainedmodelsandhyperparameteranalysisastwoapproachesatthedeeplearningalgorithmlevelcansignificantlyenhancethepointcloudclassificationusingstate-of-the-artartificialintelligencetechniques.

3Ddeeplearningcaneffectivelyhandletheclassificationoflargevolumesofpointclouddata,butthetrainingofdeeplearningalgorithmsrequireslargeamountsofannotateddata.

AnnotatedpointclouddataserveasafoundationdatabasefortheautomatedclassificationofhydraulicstructuresscannedbyLiDARusingartificialintelligencetechniques.Moreannotatedpointclouddata,whichcoverawiderrangeofhydraulicstructures,areneededforfurtherimprovingclassificationperformance.

TableofContent

TitlePage ii

Disclaimer iv

Acknowledgements v

ExecutiveSummary vi

ListofFigures ix

ListofTables x

ListofAcronyms xi

INTRODUCTION 1

Background 1

ResearchNeedDefinition 2

ResearchObjectives 2

ReportOrganization 3

LITERATUREREVIEW 4

PointCloudDatafromLiDARandSonar 4

DeepLearning 5

DeepLearningforPointCloudClassification 6

RESEARCHMETHODOLOGY 8

FieldData 8

FieldSites 8

FieldDataCollectionMethods 11

TerrestrialLidarSurvey 11

BathymetricSurvey:SonarandTotalStationDataCollection 12

3DimagereconstructionusingdroneandRGBcamera 14

SummaryofFieldDataCollected 21

DeepLearning-basedPointCloudClassification 23

ModelDesign 23

AnnotationofPointCloudData 23

Selectionof3DDeepLearningAlgorithmsforPointCloudClassification 27

Transferlearningforimproved3DDeepLearningforPointCloudClassification 27

TrainingandValidationof3DDeepLearningforPointCloudClassification 29

HyperparameterTuning:LearningRateandNumberofIterations 31

HyperparameterTuning:BlockSizeandNumberofPointsperBlock 35

ModelInferencingforthePredictionofPointCloudClassification 39

InferencingResultsofModel1fortheClassificationofBridges,Vegetation,and

Ground 39

InferencingResultsofModel2fortheClassificationofBridgeComponents 41

ScientificWorkflowsforModelAutomation 43

Pre-processingofdata 44

Post-ProcessingofData 45

SoftwareImplementation 48

FindingsandConclusions 49

Recommendations 52

ImplementationandTechnologyTransferPlan 53

References 54

Appendix 56

ListofFigures

Figure3.1.WebGISportalforthefieldsurveysintheproject(numberlabelsareIDsofsites) 10

Figure3.2.IllustrationofcollectedLiDARpointcloud 12

Figure3.3.BathymetryatSite16,PharrMillBridge,CabarrusCounty,NC 13

Figure3.4.FlightplanforunmannedaerialsystemoperationatSite#5 15

Figure3.5.Orthomosaic(with3Dinformation)generatedusingStructurefromMotiontechniqueforSite

#5 17

Figure3.6.FlightplanIforUASoperationatsite16 18

Figure3.7.FlightplanIIforUASoperationatsite16 19

Figure3.8.Orthomosaic(with3Dinformation)generatedusingSfMtechniqueforSite#16 21

Figure3.9.IllustrationofmapoffusedLidarandsonardata(Lidardataareingray;sonardataareinblue;

site#:16;PharrMillRoadsite;seeTable3.1forsiteinformation) 22

Figure3.10.Tieredspatialmodelingframeworkfordeeplearning-basedpointcloudclassificationof

hydraulicstructures 23

Figure3.11.Typologyofannotationofpointclouddataofhydraulicstructures 24

Figure3.12.Illustrationofannotatedpointcloudscollectedfromthefieldworkinthisproject 25

Figure3.13.Illustrationofannotatedpointcloudsscannedfrompreviousprojectscollectedpriortothis

study(Chen,unpublisheddata) 25

Figure3.14.Aggregationofannotatedpointclouddatafordeeplearning-basedclassification(One

annotatedscanfromSite#5;seeTable3.1forsiteinformationasneeded) 26

Figure3.15.ArchitectureofConvPointsegmentationnetworksfor3Ddeeplearning-basedpointcloud

classification 29

Figure3.16.Illustrationoflearningcurvesfortrainingandvalidationofdeeplearningalgorithmfor

Model1inresponsetonumberofiterations 32

Figure3.17.Illustrationoflearningcurvesfortrainingandvalidationofdeeplearningalgorithmfor

Model1inresponsetolearningrate 32

Figure3.18.Illustrationoflearningcurvesfortrainingandvalidationofdeeplearningalgorithmfor

Model2inresponsetolearningrate 34

Figure3.19.ResponsesurfaceofIntersectionoverUnionforModel1betweenblocksizeandnumberof

pointsperblock 36

Figure3.20.ResponsesurfaceofIntersectionoverUnionforModel2betweenblocksizeandnumberof

pointsperblock 37

Figure3.21.ComparisonbetweenannotatedpointcloudandpredictedpointcloudfromModel1 37

Figure3.22.ComparisonbetweenannotatedpointcloudandclassifiedpointcloudbyModel2 38

Figure3.23.DemonstrationofpredictionresultsofModel1trainedonbridge-vegetation-grounddataset.

. 40

Figure3.24.DemonstrationofpredictionresultsofModel2trainedonbridgecomponentdataset.A:Site#

2.D:Site#5.B,C,E,andFarefrompreviousprojects 42

Figure3.25.TheframeworkoftheDeepHydscientificworkflows(manualmodulesareingrayand

automatedmodulesareinblue) 43

Figure3.26.Inputandoutputfileformatsforscientificworkflowsofmodeltrainingandinferencefor

pointcloudclassification 45

Figure3.27.SnapshotofthemainwebpageoftheWebportalforusingDeepHydforpointcloud

classification 47

Figure3.28.Snapshotofweb-basedvisualizationofclassifiedpointcloudsinDeepHyd 47

ListofTables

Table2.1.ListofUAS-compatiblebathymetricLiDARs(SD:secchidiskdepth) 5

Table3.1.Listofsurveysitesanddatacollectedfortheproject 9

Table3.2.Summaryoffieldworkconductedfortheproject. 10

Table3.3.Summaryofdatacollectioninstruments 11

Table3.4.EstimationofaveragewateredgeelevationatSite#16(7transectswereused 14

Table3.5.ListofgroundcontrolpointsforSite#5. 16

Table3.6.Localizationaccuracypergroundcontrolpoint(unit:USSurveyfeet) 16

Table3.7.Statisticsoferrorsbetweeninitialandcomputedimagepositions 16

Table3.8.Summaryofkeypointsandmatchedkeypointsperimage 16

Table3.9.Informationoncameracalibration 17

Table3.10.ListofgroundcontrolpointsforSite#16. 19

Table3.11.Localizationaccuracypergroundcontrolpoint(std:standarddeviation;unit:USsurveyfeet).

. 20

Table3.12.Statisticsofinitialandcomputedimagepositions(std:standarddeviation) 20

Table3.13.Summaryof2Dkeypointsandmatched2Dkeypointsperimage 20

Table3.14.Informationoncameracalibration. 20

Table3.15.Listofsurveysitesanddatacollectedfortheproject(seeTable3.1forsiteinformation) 22

Table3.16.Summaryoffielddatatypescollected 22

Table3.17.Bridge-vegetation-grounddataset 26

Table3.18.Bridge-componentdataset 26

Table3.19.Accuracyperformanceofthetop5deepneuralnetworksontheSemantic3Dbenchmark 27

Table3.20.Labeled3Dpointcloudbenchmark 28

Table3.21.ListofhyperparametersfortheDeepHydsystem 30

Table3.22.SummaryofexperimentaldesignofModel1aswellasGPUcomputingperformance 33

Table3.23.SummaryofexperimentaldesignofModel2aswellascomputingtime 34

Table3.24.Hyperparametersof3DdeeplearningalgorithmforModel1. 35

Table3.25.Hyperparameterconfigurationsof3DdeeplearningalgorithmforModel2 35

Table3.26.Confusionmatrixofpointcloudclassificationusingapointcloudscanfromafieldsite(Site

#16:PharrMillsitewasused) 38

Table3.27.ConfusionmatrixofclassificationresultsbyModel2(ascanfromSite#16:PharrMillsite

wasused) 39

Table3.28.ResultsofpointcloudclassificationperformanceofModel1 39

Table3.29.ConfusionmatrixofpointcloudclassificationforModel1intermsofpercentage. 40

Table3.30.ResultsofModel2performanceforthepointcloudclassificationofbridgecomponents. 41

Table3.31.ConfusionmatrixofclassificationresultsforModel2(Wall:retainingwall) 41

Table3.32.ListoflabelsforclassesusedbytheDeepHydsystem 45

Table3.33.ListofkeysoftwareorlibrariesusedfortheimplementationofDeepHyd 48

TableA1.WeatherconditionsforUASoperationonSite#5 56

TableA2.PlanningparametersfortheflightplanforSite#5. 56

TableA3.WeatherdetailsofUASoperationonSite#16(seeTable3.1forsiteinformation) 56

TableA4.PlanningparametersfortheflightplanonSite#16. 56

TableB.1.ComputingtimeforusingDeepHydforpointcloudclassificationonsampledatasets 57

ListofAcronyms

AGL

AboveGroundLevel

AI

ArtificialIntelligence

CNN

ConvolutionalNeuralNetworks

CPU

CentralProcessingUnit

DEM

DigitalElevationModel

DSM

DigitalSurfaceModel

FAA

FederalAviationAdministration

GCP

GroundControlPoints

GIS

GeographicInformationSystem

GNSS

GlobalNavigationSatelliteSystem

GPS

GlobalPositioningSystem

GPU

GraphicsProcessingUnits

IO

InternalOrientation

IoU

IntersectionoverUnion

LiDAR

LightDetectionandRanging

MLP

Multi-layeredperception

NAD

NorthAmericaDatum

RMSE

RootMeanSquareError

RTK

Real-TimeKinematicPositioning

SfM

StructurefromMotion

Sonar

SoundNavigationandRanging

UAS

UnmannedAerialSystems

VRS

VirtualReferenceStation

PAGE

32

INTRODUCTION

Background

ThestudyofhydraulicstructuressuchasbridgesandculvertshasreceivedconsiderableattentioninparticularastheupgradeofphysicalinfrastructurehasbecomeanationalpriorityfortheUnitedStates(ASCE2021).Monitoringtheconditionofhydraulicstructuressuchasbridgesplaysapivotalroleinwarrantingthesafetyoftransportationinfrastructureandtheirsustainability.ThemonitoringofhydraulicstructuresinNChasbecomeanurgentneed,inparticularforthesystematicmanagementofNCDOT’sassets,thedevelopmentofguidelinesforroadwaydrainageandhighwaystormwatermanagement,anddocumentationofcompliancewithNCDOTandfederalstandardsfrom,e.g.,FEMAandFHWA.InNC,eachoftheState’sapproximately13,500bridgesneedstobeinspectedbyNCDOTeverytwoyearsorlesstoensuretheirstructuralstabilityandhealthforpublicsafety(NCDOT2022).Approximately8.2%oftheNCbridgesareevaluatedasinpoorcondition(byMarch2021)andinneedofimmediatemaintenance.

AsuiteoftechniquessuchasLiDARandsonar(Watsonetal.2013,BurgueraandOliver2016)havebeenextensivelyusedforthedetectionandmeasurementofhydraulicstructures.Forexample,LiDARtechniques(typicallyincludingairborne,terrestrial,andmobile)havebeenrecognizedasapowerfulandhigh-resolutionapproachforthedocumentationof3Dshapesofhydraulicstructuresandtheirsurroundingenvironments(FerozandAbuDabous2021).Atthesametime,sonartechniquescandelineate3Dcharacteristicsofunderwatertopography.Thecombinationofthesetwotechniquesprovidessupportforthemonitoringofhydraulicstructuresforbothabove-andunder-waterconditions.Theircapabilitiesinthequantificationof3Dcharacteristicsofhydraulicstructureshavebeenwellrecognized,especiallywhencomparedtotraditionalvisualinspectionmethodsthatareoftensubjectiveandlaborintensive(PrendergastandGavin2014).

TheuseofLiDARandsonartechniquesleadstothegenerationoflarge3Dpointclouddatasets.Thesepointclouddataareofgreathelpforrepresenting3Dcharacteristicsofhydraulicstructures,whichareoftenfedintohydraulicsmodelsforthein-depthinvestigationofhydraulicstructures.However,thesepointclouddataareunstructuredandtypicallyinlargevolumes.Theprocessingoftheselargepointclouddatatendstobebothlabor-andcomputation-intensive,whichposesasignificantchallengeintheclassificationofthesedata.Further,differenttypesofhydraulicstructuresexistandtheirgeometriccharacteristicsmaybesophisticatedandchangeovertime.Thisfurthercomplicatestheclassificationofthesepointclouddataforthemonitoringofhydraulicstructures.Inotherwords,theclassificationofpointclouddatacollectedfromLiDARandsonartechniquesrepresentsabigdata-drivenchallenge(TangandFeng2017).

ArtificialintelligenceholdsgreatpotentialinresolvingthechallengesassociatedwiththeclassificationofpointclouddatafromLiDARandsonar.Overthepastfewyears,artificialintelligencetechniques,representedbydeeplearning,havebeenincreasinglydevelopedandappliedtoreal-worldproblemsolving(Goodfellowetal.2016,LeCunetal.2015).Thistrend

willcontinueasartificialintelligencehasbecomeanation-widepriority.Deeplearningtechniqueshavebeenappliedtovariousstudies(e.g.,unmanneddriving,naturallanguageprocessing,remotesensing,andmedicalstudies)tosupporttheneedsofclassification,patternrecognition,andcomputervision(Guoetal.2016,Goodfellowetal.2016).Deeplearningtechniqueshavebeenhighlytoutedbecauseoftheirsuperiorperformanceoverconventionalmodelingapproaches.Theuseofdeeplearningtechniquesoftenleadstosignificantsavingsinlaborandcosts.

ResearchNeedDefinition

AccordingtoNCDOTResearchNeedStatement(RNS#:9102),theNCDOTHydraulicUnitisinterestedinutilizinghigh-resolutionLiDARandbathymetricsonardataforthedetectionandclassificationofhydraulicstructuresandtheiras-builtconditions.Thisrequirestheuseofartificialintelligencemethodstosupporttheautomatedclassificationofpointclouddataforthedetectionandevaluationofhydraulicstructures.AnAI-basedpointcloudclassificationsolutionwillbringsignificantbenefitsforNCDOTwhenLiDARandsonartechniquesareblendedandusedtofacilitatethedevelopmentofguidelinesforhighwaystormwatermanagement,roadwaydrainage,andhydraulicdesignandevaluation.

ResearchObjectives

Theoverallobjectiveofthisprojectistodevelopaspatiallyexplicit3Dmodelingframeworkandsoftwarepackagethatarebasedondeeplearningasacutting-edgeartificialintelligenceapproachforautomatedandreliableclassificationofhydraulicstructuresfrompointclouddata(DeepHyd;seeFigure1.1).Thedeeplearning-basedartificialintelligencesolution(DeepHyd,includingframeworkandsoftwaretools)canhelpresolvethechallengesassociatedwiththeextractionandclassificationofhydraulicfeaturesfromLiDARandsonardatawhilealsohavingtheflexibilityandpotentialtoincorporateadditionalmanualsurveydataandinformationfromdigitalphotography.

ToaddresstheNCDOTresearchneeds,thisprojecthasfourgoalsestablishedfortheDeepHydmodelingframework:

Goal1:tocollectacombinationofLiDAR,sonar,GPS,aerialphotometricandplanesurveydatafrom11NCDOThydraulicstructuresinCabarrus,Gaston,IredellandMecklenburgcounties,NC

Goal2:topre-processthedatacollectedfromGoal1.

Goal3:developadeeplearning-basedartificialintelligenceapproachfortheclassificationofthepointclouddataintohydraulicstructuresofinterest.

Goal4:automatethisentireclassificationeffort(training,validation,andprediction)usingscientificworkflows.

Figure1.1.DesignofDeepHyd:Adeeplearning-based3DmodelingframeworkfortheautomatedclassificationofhydraulicstructuresfromLiDARandsonardata.

ReportOrganization

Therestofthisreportisorganizedinthefollowingstructure.Section2presentsaliteraturereviewexaminingtheroleofdeeplearningmethodologiesintheclassificationoflargevolumesofpointclouddatageneratedfromLiDARandsonarscans.Section3focusesondiscussingtheresearchmethodologyemployedinthisstudy,includingfielddatacollection,deeplearning-basedpointcloudclassification(trainingandvalidationofdeeplearningalgorithms,modelinferencingorpredictionforpointcloudclassification),scientificworkflowsfortheautomationofpointcloudclassification,andsoftwareimplementation.Section4discussesfindingsandconclusionsfromtheDeepHydresearchproject.Section5presentsrecommendationsforutilizingtheDeepHydsystemandsuggestionsforfutureresearcheffortsinthisarea.

LITERATUREREVIEW

PointCloudDatafromLiDARandSonar

LiDAR(LightDetectionandRanging),alsoknownaslaserradarsystem,isanopticalremotesensingtechnologydevelopedforrangedetection(Chen2012).Bydeterminingtheheterodynelaserbeamphaseshifts,scanningLiDARcandetectthedistanceinformationfromaplaneofdatapoints,calledpointcloud.Thepointcloudinformation,whichbasicallyconsistsofthephysicalpositionsofanysurfacethatthelaser“sees”,canthenbeusedtodetectusefulcriticalinformationaboutastructureincludingtheelevation(underclearance),surfacedefects(damagequantification)anddeformationunderloading(deflectionmeasurements),etc.Contrasttoconventionalanalysisofphotographicimages,relativelysimplealgorithmscanbeusedtomanipulatethegeometricpointclouddatatoretrievetheafore-mentionedinformation.

Inearly2000