9、分布式文件系统(HDFS应用)-学习课件

Oracle·WDPe·WDP分布式文件系统（HDFS应用1）Oracle高校大数据课程系列本课目标理解HDFS框架掌握HDFS命令行操作掌握HDFSAPI操作Coursecatalogue课程目录HDFS框架1HDFS命令行操作2HDFSAPI操作3HDFS框架HDFS也是按照Master和Slave的结构。分NameNode、SecondaryNameNode、DataNode这几个角色。HDFS框架：HDFS框架NameNode：是Master节点，是大领导。管理数据块映射；处理客户端的读写请求；配置副本策略；管理HDFS的名称空间；namenode内存中存储的是=fsimage+edits。SecondaryNameNode：是一个小弟，分担大哥namenode的工作量；是NameNode的冷备份；合并fsimage和fsedits然后再发给namenode。DataNode：Slave节点，奴隶，干活的。负责存储client发来的数据块block；执行数据块的读写操作。fsimage:元数据镜像文件（文件系统的目录树）edits：元数据的操作日志（针对文件系统做的修改操作记录）HDFS框架：Coursecatalogue课程目录HDFS框架1HDFS命令行操作2HDFSAPI操作3HDFSSHELL

使用这些命令，首先学会help的使用是很明智的选择。它的应用语法在一个已经启动Hadoop的HDFS服务与YARN服务的集群中，用Hadoop关键字加上“-help”即可列出所有HadoopShell支持的命令。HDFSSHELL：-help[user@master~]$hadoop-helpUsage:hadoop[--configconfdir]COMMANDwhereCOMMANDisoneof:fsrunagenericfilesystemuserclientversionprinttheversionjar<jar>runajarfilechecknative[-a|-h]checknativehadoopandcompressionlibrariesavailabilitydistcp<srcurl><desturl>copyfileordirectoriesrecursivelyarchive-archiveNameNAME-p<parentpath><src>*<dest>createahadooparchiveclasspathprintstheclasspathneededtogetthecredentialinteractwithcredentialprovidersHadoopjarandtherequiredlibrariesdaemonlogget/settheloglevelforeachdaemontraceviewandmodifyHadooptracingsettingsorCLASSNAMEruntheclassnamedCLASSNAMEHDFSSHELL通过hadoop–help查询列出所有fs支持的语法规则HDFSSHELL：-help[user@master~]$hadoopfs-helpUsage:hadoopfs[genericoptions][-appendToFile<localsrc>...<dst>][-cat[-ignoreCrc]<src>...][-checksum<src>...][-chgrp[-R]GROUPPATH...][-chmod[-R]<MODE[,MODE]...|OCTALMODE>PATH...][-chown[-R][OWNER][:[GROUP]]PATH...][-copyFromLocal[-f][-p][-l]<localsrc>...<dst>][-copyToLocal[-p][-ignoreCrc][-crc]<src>...<localdst>][-count[-q][-h]<path>...][-cp[-f][-p|-p[topax]]<src>...<dst>][-createSnapshot<snapshotDir>[<snapshotName>]][-deleteSnapshot<snapshotDir><snapshotName>][-df[-h][<path>...]][-du[-s][-h]<path>...][-expunge]…….HDFSSHELL通过hadoop–help查询列出所有fs支持的语法规则HDFSSHELL：-help[user@master~]$hadoopfs-helpUsage:hadoopfs[genericoptions][-appendToFile<localsrc>...<dst>][-cat[-ignoreCrc]<src>...][-checksum<src>...][-chgrp[-R]GROUPPATH...][-chmod[-R]<MODE[,MODE]...|OCTALMODE>PATH...][-chown[-R][OWNER][:[GROUP]]PATH...][-copyFromLocal[-f][-p][-l]<localsrc>...<dst>][-copyToLocal[-p][-ignoreCrc][-crc]<src>...<localdst>][-count[-q][-h]<path>...][-cp[-f][-p|-p[topax]]<src>...<dst>][-createSnapshot<snapshotDir>[<snapshotName>]][-deleteSnapshot<snapshotDir><snapshotName>][-df[-h][<path>...]][-du[-s][-h]<path>...][-expunge]…….lshadoopfs–ls/列出hdfs文件系统根目录下的目录和文件hadoopfs–ls–R/列出hdfs文件系统所有的目录和文件puthadoopfs-put<localfile><hdfsfile>从本地文件系统中复制单个或多个源路径到目标文件系统。也支持从标准输入中读取输入写入目标文件系统。hadoopfs–ls/experiment/datashadoopfs–put/home/admin/experiment/datas/test.txt/experiment/datashadoopfs–ls/experiment/datasgethadoopfs–get<hdfsfile><localfileordir>复制文件到本地文件系统。ls/home/admin/experiment/datashadoopfs–get/experiment/datas/test.txt/home/admin/experiment/datasls/home/admin/experiment/datasrmhadoopfs-rm<hdfsfile>删除指定的文件。只删除非空目录和文件。hadoopfs–ls/experiment/datashadoopfs–rm/experiment/datas/test.txtrmhadoopfs–ls/experiment/datasmkdirhadoopfs-mkdir<hdfspath>只能一级一级的建目录，父目录不存在的话使用这个命令会报错hadoopfs–ls/hadoopfs–mkdir/hdfsmkdirhadoopfs–ls/mkdirhadoopfs-mkdir–p<hdfspath>所创建的目录如果父目录不存在就创建该父目录hadoopfs–ls/hadoopfs–mkdir–p/hdfsmkdirhadoopfs–ls/hadoopfs–ls/hdfscphadoopfs-cp<hdfsfile><hdfsfile>将文件从源路径复制到目标路径。这个命令允许有多个源路径，此时目标路径必须是一个目录。hadoopfs–ls/hdfs/hdfs_shellhadoopfs–cp/experiment/datas/test.txt/hdfs/hdfs_shell/test.txtcphadoopfs–ls/hdfs/hdfs_shellmvhadoopfs-mv<hdfsfile><hdfsfile>目标文件不能存在，否则命令不能执行，相当于给文件重命名并保存，源文件不存在hadoopfs–ls/experiment/datashadoopfs–mv/experiment/datas/test.txt/hdfs/hdfs_shellmvhadoopfs–ls/experiment/datashadoopfs–ls/hdfs/hdfs_shellfsckhadoopfsck<path>[-move|-delete|-openforwrite][-files[-blocks[-locations|-racks]]]运行HDFS文件系统检查工具hadoopfsck/experiment/datas-filesbalancerhadoopbalancer[-threshold<threshold>]运行集群平衡工具。管理员可以简单的按Ctrl-C来停止平衡过程。hadoopbalancerdfsadminhadoopdfsadmin[-report][-safemodeenter|leave|get|wait][-refreshNodes][-finalizeUpgrade][-upgradeProgressstatus|details|force][-metasavefilename][-setQuota<quota><dirname>...<dirname>][-clrQuota<dirname>...<dirname>][-help[cmd]]运行一个HDFS的dfsadmin客户端。hadoopdfsadmin-reportCoursecatalogue课程目录HDFS框架1HDFS命令行操作2HDFSAPI操作3HDFSAPI创建目录：获得FileSystem对象，根据提供的API进行相应操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***创建文件夹**@paramfolder文件夹名*/publicstaticvoidmkdirs(Stringfolder)throwsIOException{Pathpath=newPath(folder);FileSystemfs=FileSystem.get(URI.create(HDFS),conf);if(!fs.exists(path)){fs.mkdirs(path);System.out.println("Create:"+folder);}fs.close();}HDFSAPI删除目录：获得FileSystem对象，根据提供的API进行相应操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***删除文件夹**@paramfolder文件夹名*/publicstaticvoidrmr(Stringfolder)throwsIOException{Pathpath=newPath(folder);FileSystemfs=FileSystem.get(URI.create(HDFS),conf);fs.deleteOnExit(path);System.out.println("Delete:"+folder);fs.close();}HDFSAPI重命名文件：获得FileSystem对象，根据提供的API进行相应操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***重命名文件*@paramsrc源文件名*@paramdst目标文件名**/publicstaticvoidrename(Stringsrc,Stringdst)throwsIOException{Pathname1=newPath(src);Pathname2=newPath(dst);FileSystemfs=FileSystem.get(URI.create(HDFS),conf);fs.rename(name1,name2);System.out.println("Rename:from"+src+"to"+dst);fs.close();}HDFSAPI列出目录下的文件信息：获得FileSystem对象，根据提供的API进行相应操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***列出该路径的文件信息*@paramfolder文件夹名*/publicstaticvoidls(Stringfolder)throwsIOException{Pathpath=newPath(folder);FileSystemfs=FileSystem.get(URI.create(HDFS),conf);FileStatus[]list=fs.listStatus(path);System.out.println("ls:"+folder);System.out.println("=======================");for(FileStatusf:list){System.out.printf("name:%s,folder:%s,size:%d\n",f.getPath(),f.isDirectory(),f.getLen());}System.out.println("===============");fs.close();}HDFSAPI创建文件：获得FileSystem对象，根据提供的API进行相应操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();publicstaticvoidcreateFile(Stringfile,Stringcontent)throwsIOException{FileSystemfs=FileSystem.get(URI.create(HDFS),conf);byte[]buff=content.getBytes();FSDataOutputStreamos=null;try{os=fs.create(newPath(file));os.write(buff,0,buff.length);System.out.println("Create:"+file);}finally{if(os!=null)os.close();}fs.close();}HDFSAPI上传本地文件到HDFS：获得FileSystem对象，根据提供的API进行相应操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***上传本地文件到hdfs**@paramlocal本地文件路径*@paramremotehdfs目标路径*/publicstaticvoidcopyFile(Stringlocal,Stringremote)throwsIOException{FileSystemfs=FileSystem.get(URI.create(HDFS),conf);fs.copyFromLocalFile(newPath(local),newPath(remote));System.out.println("copyfrom:"+local+"to"+remote);fs.close();}HDFSAPI下载HDFS文件到本地：获得FileSystem对象，根据提供的API进行相应操作privatestaticfinalStringHDFS="hdfs://master:9000/";privatestaticfinalConfigurationconf=newConfiguration();/***从hdfs下载文件到本地**@paramremotehdfs文件路径*@paramlocal本地目标路径*/publicstaticvoiddownload(Stringremote,