Jetson 平台交叉编译开发与实现

JETSON

PLATFORMJetson

SA:

Alan

ZhangApril

24，20202Jetson

product

andmarketingWhy

weneed

crosscompileHow

to

setup

the

cross-compile

environmentConfig

directly

via

the

environment

variable.Config

in

Makefile.Build

your

simulation.Compile

on

X86

host

and

execute

on

JetsonCross

compile

Linux

kernel.Cross

compile

CUDA

project.Cross

compile

Multi

Media

API

based

application.Cross

compile

Deepstream.Cross

compile

CaffeCross

compile

OpenCV.Where

canI

get

the

prebuild

version

and

how

to

build

onjetson？AGENDA3NVIDIA

JETSONSoftware-Defined

AI

PlatformAI

atthe

EdgeSensor

Fusion

&

Compute

PerformanceECOSYSTEMExpertise,

Time

to

MarketJETSON

COMPUTERSOFTWARE

DEFINEDSDK,

Design

Tools,

Libs,

GEMsJetpack

SDK

∙

CUDA

∙

TensorRT

∙

TensorFlow

∙

ONMX

∙

ROSArtificial

IntelligenceComputer

VisionAcceleratedComputingMultimediaGesture

recObj

detectPath

planningDepth

estPose

estSpeech

recActSenseReasonINTELLIGENT

VIDEO

ANALYTICS

FOREFFICIENCY

AND

SAFETYJETSON

ROBOTICS

&

LOGISTICIndustrialAerospace/DefenseHealthcareConstructionAgricultureSmart

CityInspectionServiceRetailLogisticsCollaborationDeliveryTHE

JETSON

FAMILYfor

AI

at

the

Edge

and

Autonomous

SystemdesignsSame

softwareListed

prices

are

for

1000u+

|

Full

specs

at

/jetson*

TX2i:

10-20W7.5

–

15W*50mm

x

87mmJETSON

TX2series1.3

TFLOPS

(FP16)5

-

10W45mm

x

70mmJETSON

NANO0.5

TFLOPS

(FP16)10

–

30W100mm

x

87mmJETSON

AGXXAVIER

series11

TFLOPS

(FP16)32

TOPS(INT8)10

-

15W45mm

x

70mmJETSON

Xavier

NX6

TFLOPS

(FP16)21

TOPS(INT8)AI

at

the

edgeFully

autonomous

machinesJetson

AGX

Xavier

DeveloperKitDeveloper

Kit

User

GuideL4T

DocumentationModule

DatasheetOEM

Design

GuideJetson

TX2

DeveloperKitDeveloper

Kit

User

GuideL4T

DocumentationModule

DatasheetOEM

Design

GuideJetson

Nano

DeveloperKitGetting

StartedDeveloper

Kit

User

GuideL4T

DocumentationJETSONNANOJETSON

TX2JETSON

XAVIER

NXJETSON

AGX

XAVIERGPU128

Core

Maxwell0.5

TFLOPs

(FP16)256

Core

Pascal1.3

TFLOPS

(FP16)384

Core

Volta21

TOPs

(INT8)512

Core

Volta

+

NVDLA10

TFLOPS

(FP16)32

TOPS

(INT8)CPU4

core

ARM

A576

coreDenver

and

A57(2x)

2MB

L26

core

Carmel

ARM

CPU(3x)

2MB

L2

+

4MB

L38

coreCarmel

ARM

CPU(4x)

2MB

L2

+

4MB

L3Memory4

GB64-bit

LPDDR425.6

GB/sUp

to

8

GB128b

LPDDR458

GB/s8

GB128-bit

LPDDR4x51.2

GB/sUp

to

32GB

256-bitLPDDR4x137

GB/sStorage16

GBeMMCUp

to

32

GBeMMC16

GBeMMC32

GBeMMCEncode4K

@30

(H.265)4K

@

60

(H.265)2x

4K

@

30(H.265)4x

4K

@

60(H.265)Decode4K

@

60

(H.265)2x

4K

@

60

(H.265)2x

4K

@

60

(H.265)6x

4K

@

60(H.265)Camera12

(3x4

or

4x2)

MIPI

CSI-2

D-PHY

1.1

lanes

(18Gbps)12

lanes

MIPICSI-2D-PHY

1.2

(30

Gbps)C-PHY

(41

Gbps)12

lanes

(3x4

or

6x2)MIPI

CSI-2D-PHY

1.2

(30

Gbps)16

lanes

MIPI

CSI-2

|

8lanes

SLVS-ECD-PHY

(40

Gbps)C-PHY

(59

Gbps)Mechanical69.6mm

x

45mm260

pin

edge

connector87mm

x

50mm400

pin

connector69.6mm

x

45mm260

pin

edge

connector100mm

x

87mm699

pin

connectorSoftwareJetPack

SDK

–

Unified

software

release

across

all

Jetson

productsJETSONSOFTWARENsight

Developer

ToolsJetsonDeepStream

&

Isaac

SDKsModulesDepthestimationPathplanningObjectdetectionGesturerecognitionEcosystemmodulesPoseestimation…CUDA

•

Linux

•

RTOSlibargusVideo

APIMultimediaVisionWorksOpenCVComputer

VisionTensorRTcuDNNDeep

LearningSensorsDriversEcosystemAccelerated

ComputingcuBLAScuFFTJetson

software:

/jetsonJetPack

SDKCUDA-X10Jetson

product

andmarketingWhy

weneed

crosscompileHow

to

setup

the

cross-compile

environmentConfig

directly

via

the

environment

variable.Config

in

Makefile.Build

your

simulation.Compile

on

X86

host

and

execute

on

JetsonCross

compile

Linux

kernel.Cross

compile

CUDA

project.Cross

compile

Multi

Media

API

based

application.Cross

compile

Deepstream.Cross

compile

CaffeCross

compile

OpenCV.Where

canI

get

the

prebuild

version

and

how

to

build

onjetson？AGENDAARCHITECTURES

&

INSTRUCTION

SETSHow

compile

works11ARCHITECTURES

&

INSTRUCTION

SETSNative

compile

and

cross

copile12ARCHITECTURES

&

INSTRUCTION

SETS13What

can

we

benefit

from

crosscompileSpeedTarget

platform

usually

designed

to

have

low

cost

and

low

power

consumption.

So

the

speed

ofthe

CPU

will

be

not

good

at

compiling.CapabilityTarget

platform

have

less

storage

and

less

memory

to

sustain

running

a

big

project.AvailabilityNot

all

the

system

for

target

platform

has

the

resources

to

support

compilenatively.FlexibilityA

complete

build

environment

need

lots

of

the

software

to

support.14Jetson

product

andmarketingWhy

weneed

crosscompileHow

to

setup

the

cross-compile

environmentConfig

directly

via

the

environment

variable.Config

in

Makefile.Build

your

simulation.Compile

on

X86

host

and

execute

on

JetsonCross

compile

Linux

kernel.Cross

compile

CUDA

project.Cross

compile

Multi

Media

API

based

application.Cross

compile

Deepstream.Cross

compile

CaffeCross

compile

OpenCV.Where

canI

get

the

prebuild

version

and

how

to

build

onjetson？AGENDANVIDIA®

specifies

the

Linaro®

gcc

7.3.1

2018.05

aarch64

toolchain

for:Cross-compiling

applications

torun

onJetson

Linux

Driver

Package

(L4T

rel-32.Cross-compiling

code

in

the

L4T

rel-32

source

release.This

topic

describes

how

toobtain

this

toolchain.ToolchainInformationThe

toolchain

contains

the

following

components:GCC

version:

7.3.1Binutils

version:

0170706Glibc

version:

2.25Downloading

the

ToolchainDownload

the

pre-built

toolchain

binaries

from:/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-

x86_64_aarch64-linux-gnu.tar.xzExtracting

the

ToolchainExecute

the

following

commands

to

extract

thetoolchain:$

mkdir

$HOME/l4t-gcc$cd

$HOME/l4t-gcc$

tar

xf

gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xzSETUP

THE

CROSS

COMPILEDowload

the

toolchainSETUP

THE

CROSS

COMPILE16export

CROSS_ROOT=/home/alan/rootfs-nanoexport

CROSS_COMPILE

=

/home/alan/toolchain/gcc-4.8.5-aarch64/binexport

CC

=

${CROSS_COMPILE}/aarch64-unknown-linux-gnu-gccexport

CXX

=

${CROSS_COMPILE}/aarch64-unknown-linux-gnu-g++export

LD

=

${CROSS_COMPILE}/aarch64-unknown-linux-gnu-ldexport

AR

=

${CROSS_COMPILE}/aarch64-unknown-linux-gnu-arexport

AS

=

${CROSS_COMPILE}/aarch64-unknown-linux-gnu-asexport

RANLIB

=

${CROSS_COMPILE}/aarch64-unknown-linux-gnu-ranlibexport

NVCC

=${CROSS_ROOT}/usr/local/cuda/bin/nvccexport

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SYSROOT/usr/lib/aarch64-linux-gnu:$SYSROOT/lib/aarch64-linux-gnu:$SYSROOT/libexport

LC_ALL=Cexport

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/alan/toolchain/gcc-4.8.5-aarch64/aarch64-unknown-linux-gnu/lib64:/home/alan/toolchain/gcc-4.8.5-aarch64/aarch64-unknown-linux-gnu/sysroot/libOptions

1:Options

2:17$

sudo

apt

update$

sudoapt

installnfs-kernel-server$

sudo

vim/etc/exports/

*(rw,sync,no_subtree_check,

no_root_squash)$

sudo

exportfs

–a$

sudo

systemctl

restart

nfs-kernel-server.service$

sudoexportfs/

(world)18Setupthe

NFS

on

Jetson$

sudo

apt

update$

sudo

apt

install

nfs-common$

sudo

mkdir

–p/mnt/rootfs$

sudo

/etc/fstab:/

/mnt/rootfs

nfs

defaults

0

0$

sudo

mount

–t

nfs

:/

/mnt/rootfs$

ls/mnt/rootfs/Setupthe

NFS

on

Host19Chroot

&

Qeum•••Options

3:20Chroot

&

Qeum/2122Jetson

product

andmarketingWhy

weneed

crosscompileHow

to

setup

the

cross-compile

environmentConfig

directly

via

the

environment

variable.Config

in

Makefile.Build

your

simulation.Compile

on

X86

host

and

execute

on

JetsonCross

compile

Linux

kernel.Cross

compile

CUDA

project.Cross

compile

Multi

Media

API

based

application.Cross

compile

Deepstream.Cross

compile

CaffeCross

compile

OpenCV.Where

canI

get

the

prebuild

version

and

how

to

build

onjetson？AGENDA/jetson/l4t-multimedia/cross_platform_support.htmlCross

compile

the

Kernel$

green@C

~/nvidia/nvidia_sdk/sources/kernel/kernel-4.9

$

git

tag

–l/jetson/l4t/index.htmlCross

compile

the

Kernel$

green@C

~/nvidia/nvidia_sdk/sources/kernel/kernel-4.9

$

./source_sync.sh

-t

tegra-l4t-r32.3.1/jetson/l4t/index.htmlCross

compile

the

Kernelgreen@C

~/nvidia/nvidia_sdk/sources

$

lshardware

kernel

u-bootgreen@C

~/nvidia/nvidia_sdk/sources

$

cd

kernel/kernel-4.9/green@C

~/nvidia/nvidia_sdk/sources/kernel/kernel-4.9

$export

ARCH=arm64export

CROSS_COMPILE=${HOME}/Downloads/toolchain/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-export

LOCALVERSION=-tegranvidia@nvidia-desktop:/usr/local/cuda/samples$

uname

-aLinux

nvidia-desktop

4.9.140-tegra

#1

SMP

PREEMPT

Mon

Dec

9

22:52:02

PST

2019

aarch64

aarch64

aarch64

GNU/Linux$

wget

/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-

7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xzCross

compile

the

KernelJetson:nvidia@nvidia-desktop:/usr/local/cuda/samples$sudo

cp

/proc/config.gz

/[sudo]

password

for

nvidia:nvidia@nvidia-desktop:/usr/local/cuda/samples$X86Host:green@C

~/nvidia/nvidia_sdk/sources/kernel/kernel-4.9$

sudo

gzip

-d

/mnt/rootfs/config.gzgreen@C

~/nvidia/nvidia_sdk/sources/kernel/kernel-4.9

$

cp

/mnt/rootfs/config

.'/mnt/rootfs/config'

->

'./config'removed

'./config'green@C

~/nvidia/nvidia_sdk/sources/kernel/kernel-4.9$

cp

config

.config'config'

->'.config'green@C

~/nvidia/nvidia_sdk/sources/kernel/kernel-4.9

$

make

menuconfigscripts/kconfig/mconf

KconfigMake–j8/jetson/l4t/index.htmlGet

the

config

fileKERNEL

CROSS

COMPILE28green@C

~/nvidia/nvidia_sdk/sources/kernel/kernel-4.9

$

cp

arch/arm64/boot/Image

/mnt/rootfs/boot/export

TARGET_ROOTFS=/mnt/rootfssudo

ln

-sf

/mnt/rootfs/usr/lib/aarch64-linux-gnu

/usr/lib/aarch64-linux-gnusudo

ln

-sf

/usr/lib/aarch64-linux-gnu/libpthread.so

/lib/libpthread.so.0sudo

ln

-sf

/mnt/rootfs/lib/aarch64-linux-gnu

/lib/aarch64-linux-gnusudo

ln

-sf

/mnt/rootfs/lib/aarch64-linux-gnu/libc.so.6

/lib/libc.so.6sudo

ln-sf

/mnt/rootfs/usr/lib/aarch64-linux-gnu/libc_nonshared.a

/usr/lib/libc_nonshared.asudo

ln

-sf

/lib/aarch64-linux-gnu/ld-2.27.so/lib/ld-linux-aarch64.so.129Cross

compile

Multi

Media

API/nvidia-jetson-linux-multimediaapireferencegreen@C

~/Downloads/nvidia/sdkm_downloads/tegra_multimedia_api

$

git

diffdiff

--git

a/samples/Rules.mk

b/samples/Rules.mkindex

7977ab1..6dd362d

100644

a/samples/Rules.mk+++

b/samples/Rules.mk@@

-76,7

+76,7

@@

NM =

$(AT)

$(CROSS_COMPILE)nm30=

$(AT)

$(CROSS_COMPILE)strip=

$(AT)

$(CROSS_COMPILE)objcopy=

$(AT)

$(CROSS_COMPILE)objdumpSTRIPOBJCOPYOBJDUMP-NVCC+NVCC=

$(AT)

$(CUDA_PATH)/bin/nvcc-ccbin

$(filter-out

$(AT),

$(CPP))=

$(AT)

$(TARGET_ROOTFS)$(CUDA_PATH)/bin/nvcc-ccbin

$(filter-out

$(AT),

$(CPP))#

Specify

the

logical

root

directory

for

headers

andlibraries.ifneq

($(TARGET_ROOTFS),)@@

-96,7

+96,8

@@

CPPFLAGS

+=

-std=c++11

\-I"$(ALGO_TRT_DIR)"

\-I"$(TARGET_ROOTFS)/$(CUDA_PATH)/include"

\-I"$(TARGET_ROOTFS)/usr/include/$(TEGRA_ARMABI)"

\++- -I"$(TARGET_ROOTFS)/usr/include/libdrm"-I"$(TARGET_ROOTFS)/usr/include/libdrm"

\-I"$(TARGET_ROOTFS)/usr/include/opencv4"#

All

common

dependent

librariesLDFLAGS

+=

\/nvidia-jetson-linux-multimediaapireferenceCross

compile

Multi

Media

APICross

compile

Multi

MediaAPI/nvidia-jetson-linux-multimediaapireference31Torun$

./video_dec_trt2

../../data/Video/sample_outdoor_car_1080p_10fps.h264../../data/Video/sample_outdoor_car_1080p_10fps.h264

H264 --trt-deployfile../../data/Model/resnet10/totxt --trt-modelfile

../../data/Model/resnet10/resnet10.caffemodelmode0--trt-Cross

compile

Multi

MediaAPI/nvidia-jetson-linux-multimediaapireference32Example$

./backend

1

../../data/Video/sample_outdoor_car_1080p_10fps.h264H264 --trt-deployfile../../data/Model/GoogleNet_one_class/GoogleNet_modified_oneClass_halfHD.prototxt --trt-modelfile../../data/Model/GoogleNet_one_class/GoogleNet_modified_oneClass_halfHD.caffemodel

--trt-mode

0

--trt-proc-interval

1

-fps

10Cross

compile

Multi

MediaAPI/nvidia-jetson-linux-multimediaapireference33export

TARGET_ROOTFS=/mnt/rootfsexport

CUDA_VER=10.0Cross

compile

Deepstream34Copy

the

model's

label

file"ssd_coco_labels.txt"

fromthe

data/ssd

directoryin

TensorRT

samples

to

this

directory.Steps

to

generate

the

UFF

model

from

ssd_inception_v2_coco

TensorFlowfrozengraph.

These

steps

have

been

referred

from

TensorRT

sampleUffSSD

README:Make

sure

TensorRT's

uff-converter-tf

package

is

installed.Installtensorflow-gpu

package

for

python:For

dGPU:$pip

installtensorflow-gpuFor

Jetson,

refer

to

/Jetson_Zoo#TensorFlowDownload

and

untar

the

ssd_inception_v2_coco

TensorFlow

trained

model

from/models/object_detection/ssd_inception_v2_coco_2017_11_17.tar.gz354.

Navigate

to

the

extracted

directory

and

convert

the

frozen

graph

to

uff:$

cdssd_inception_v2_coco_2017_11_17$

python

/usr/lib/python2.7/dist-packages/uff/bin/convert_to_uff.py

\frozen_inference_graph.pb

-O

NMS

\-p

/usr/src/tensorrt/samples/sampleUffSSD/config.py

\-o

sample_ssd_relu6.uff5.

Copy

sample_ssd_relu6.uff

to

thisdirectory.Pre-requisites

for

SSDOn

the

Host:#cd

/mnt/rootfs/home/nvidia/deepstream/deepstream-4.0/sources/objectDetector_SSD/nvdsinfer_custom_impl_ssd/ssd_inception_v2_coco_2017_11_17#

python

/usr/lib/python3.6/dist-packages/uff/bin/convert_to_uff.py/mnt/rootfs/home/nvidia/deepstream/deepstream-4.0/sources/objectDetector_SSD/nvdsinfer_custom_impl_ssd/ssd_inception_v2_coco_2017_11_17/frozen_inference_graph.pb

-O

NMS

-p

/usr/src/tensorrt/samples/sampleUffSSD/config.py-o/mnt/rootfs/home/nvidia/deepstream/deepstream-4.0/sources/objectDetector_SSD/sample_ssd_relu6.uff#

cd

/mnt/rootfs/home/nvidia/deepstream/deepstream-4.0/sources/objectDetector_SSD/#

cp

/usr/src/tensorrt/data/ssd/ssd_coco_labels.txt

.On

the

Jetson:#

export

DISPLAY=:0#

deepstream-app

-c

deepstream_app_config_ssd.txt36Cross

compile

DeepstreamCross

compile

DeepStream37Cross

compile

DeepStream#

cd/mnt/rootfs/home/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/deepstream-segmentation-test#

make38Cross

compile

DeepStreamOn

Jetson#

./deepstream-segmentation-appdstest_segmentation_config_semantic.txtsample_720p.mjpeg

sample_720p.mjpeg39SETUP

THE

CROSS

COMPILE40export

TARGET_ROOTFS=/mnt/rootfsexport

CROSS_COMPILE=${HOME}/Downloads/toolchain/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-export

CC

=${CROSS_COMPILE}/gccexport

CXX

=${CROSS_COMPILE}/g++export

LD

=

${CROSS_COMPILE}/ldexport

AR

=

${CROSS_COMPILE}/arexport

AS

=

${CROSS_COMPILE}/asexport

RANLIB

=

${CROSS_COMPILE}/ranlibexport

NVCC

=${TARGET_ROOTFS}/usr/local/cuda/bin/nvccexport

LC_ALL=COption

1:SETUP

THE

CROSS

COMPILE41for

i

in

/dev

/dev/pts

/proc

/sys

/run;

do

sudo

mount-B

$i

/mnt$i;

donesudo

mount

-t

proc

/proc

procsudo

mount

--rbind

/sys

syssudo

mount

--rbind/dev

devsudo

mount

--rbind

/runrunsudo

mount

-o

bind/dev

devsudo

mount

-o

bind

/sys

syssudo

mount-t

proc

/procprocsudo

mount

-o

bind/run

runSudo

rm

/etc/resolv.conf

echo

'nameserver

'

|

sudo

tee-a

/etc/resolv.confOption

2:SETUP

THE

QEMU42for

i

in

/dev

/dev/pts

/proc

/sys

/run;

do

sudo

mount-B

$i

/mnt$i;

donesudo

mount

-o

bind/dev

devsudo

mount

-o

bind

/sys

syssudo

mount-t

proc

/proc

procsudo

mount

-o

bind/run

runsudo

chroot

./zhj-buffer/Cross-Compile-Jetson$

cd

/mnt/rootfs/usr/local/cuda/bin/$

./cuda-install-samples-10.0.sh

~/$

cd

~/NVIDIA_CUDA-10.0_Samples$

make-j8$ cp

bin/aarch64/linux/release/

/mnt/rootfs/home/nvidia/

-rfCUDA

Cross

Compile

&

Demo43Verify

on

Jetson44DeviceQuery45$cd/usr/local/cuda/samples/1_Utilities/deviceQuery$

sudomake$

./deviceQueryCUDA

Capability

7.2$

nvidia@nvidia-desktop:/usr/local/cuda/samples$

~/release/smokeParticlesCUDA

Cross

Compile

&

Demo46Pre-request

for

Caffewget

/Kitware/CMake/releases/download/v3.17.1/cmake-3.17.1.tar.gz

apt-get

install

libssl-dev./configure47Make–j16Make

install/usr/local/bin/cmake/zhj-buffer/Cross-Compile-Jetson/installation.htmlCompile

Caffe

&

Testgit

clone

/BVLC/caffe.git/download/

sudo

apt

install

libatlas-base-devsudo

apt

installlibatlas-devsudoapt

installlibopenblas-devsudo

apt

install

libsnappy-devsudoapt

installlibleveldb-devsudo

apt

installliblmdb-dev

apt

install

libboost-all-devapt

install

libgflags-devapt

install

libgoogle-glog-dev48sudo

rm

/etc/resolv.conf

echo

'nameserver

'

|sudo

tee

-a/etc/resolv.conf/zhj-buffer/Cross-Compile-Jetson/installation.htmlPre-request

for

CaffeCompile

Caffe

&

TestHost:cd

caffevim../cmake/Cuda.cmakeset(Caffe_known_gpu_archs

"72")change

CV_LOAD_IMAGE_COLOR

to

cv::IMREAD_COLORchange

CV_LOAD_IMAGE_GRAYSCALE

to

cv::IMREAD_GRAYSCALEmkdir

build/usr/local/bin/cmake

../makeallmake

testmake

pycaffecd

/media/green/M0/rootfs-xavier/home/nvidia/caffe/buildsudo

cp

lib

/mnt/rootfs/home/nvidia/caffe/build/-rfsudo

cp

../python/

/mnt/rootfs/home/nvidia/caffe/

-rf49Jetson:export

PYTHONPATH=/home/nvidia/caffe/python:$PYTHONPATHPythonImport

caffeCompile

Caffe

&

Test50Compile

Caffe

&

test51Compile

Caffe

&

Test52Jetson:cd

/media/green/M0/rootfs-xavier/home/nvidia/caffesudocp

build/tools/

/mnt/rootfs/home/nvidia/caffe/build/-rfsud