现代数据中心解决方案

1、现代数据中心解决方案提纲思科数据中心产品介绍典型组网方案成功案例LAN/SANLANSAN思科数据中心产品系列CISCO NX-OS: From Hypervisor to Core CISCO DCNM: Single Pane of ManagementDELIVERING TO YOUR DATA CENTER NEEDSResilient, High Performance, Scalable Fabric Operational Efficiency/ ConsistencyP-V-CWorkload Mobility Within/ Across DCsSecure Separat

2、ion/ MultitenancyLAN+SAN ConvergenceCisco MDS 9200 Series Cisco MDS 9500 Series Cisco Nexus 1000VCisco Nexus 1010CiscoNexus 4000Cisco MDS 9100 SeriesCiscoNexus 5000CiscoNexus 3000CiscoNexus 2000CiscoNexus 7000CiscoNexus 6000Nexus 7000系列Nexus 7004Nexus 7009Nexus 7010Nexus 7018Target DeploymentsSmall

3、and Medium DCIdeal migration from Catalyst 6509Ideal for cold aisle - hot aisleHighest 10G DensityBandwidth Per Solt550Gbps550Gbps550Gbps550GbpsMax 10GE Density(Non-Blocking)96 ports336 ports384 ports768 portsFootprint7RU14RU21RU25RU8,800+Customers5 Million+Ports Shipped43,000 Chassis Shipped Nexus

4、7000常用接口板卡F2e 48 ports 1 and 10GE (SFP+) F2e 48 ports 1 and 10GBaseT (RJ45) N7K-F306CF-25N7K-F312FQ-25Nexus 7700系列Nexus 7718Nexus 771018-slot chassis1.3Tbps/slot16 x payload modules2 x supervisor modules6 x fabric modules3 x fan traysFront-to-back airflow16 x 3000W PSUs10-slot chassis1.3Tbps/slot8 x

5、 payload modules2 x supervisor modules6 x fabric modules3 x fan traysFront-to-back airflow8 x 3000W PSUsNexus 7710 ChassisRedundant Supervisor Engines8 payload slots (1.3T/slot)8 3000W AC/DC power supplies14RU3 fan trays6 fabric modules (behind fan trays)RearFrontFront-to-back airflow32”Nexus 7700 主

6、要业务板卡Nexus 7700 F2e 48p 10GNexus 7700 F3 24p 40GNexus 7700 F3 12p 100GNexus 7000 / Nexus 7700 ComparisonNexus 7000Nexus 7700Max System Switching Capacity 17.6+ Tbps42+ Tbps (Fab2)84+ Tbps (future)Per-Slot Bandwidth 550 Gbps1.32 Tbps (Fab2)2.64 Tbps (future)Bandwidth per fabric46Gbps/payload slot (Fa

7、b1)110Gbps/payload slot (Fab2)220Gbps/payload slot (Fab2)440Gbps/payload slot (future)Max Port Density (10G/40G/100G)768/192/96768*/384/192 (F2E/F3)Air FlowFront-to-backSide-to-sideSide-to-backFront-to-backModule typesM1/M2F1/F2/F2E/F3Nexus 7700 F2E/F3FabricsUp to 5 X Fab1Up to 5 X Fab2Up to 6 X Nex

8、us 7700 Fab2* Higher density 10G possible via 40G/100G modules using breakout cables in future48/96 port 4 slot (Q2 CY14)8 slot16 slot (Q2 CY14)高度2/3 RU6-7 RU13 RU21 RU业务板卡槽位1 GEM4816单系统交换容量 (Tbps)NA15 Tbps30 Tbps60 Tbps最大线速的10G端口数量485761152Future最大线速的40G端口数量12144288576可升级到ACI矩阵模式N9K 产品系列Nexus 9396P

9、XNexus 93128TXNexus 9504 Nexus 9508 Nexus 9516 Nexus M12PQNexus 6004PerformanceHighest density line rate, low power per port 10G/40G Ethernet/FCoE switchin the industry!Integrated Layer 2 and Layer 3 feature-setFirst 40G FCoE Switch VersatilityWide range of deployment options High density FEX aggreg

10、atorLarge Scale FabricsLeaf-Spine ArchitectureLatencyLow Latency at scale 1 microsecond latency for all ports, cut through switchingFlexibilityConfigurable port speeds with true 40GE flows or 4 x 10GE flows Modularity provides future expansion capabilitiesEnables High Performance Networking through

11、Cisco Custom SiliconForm-factor / Power4RU; 17.5” W x 30” D96 Line rate 40G ports or 384 Line rate 10G portsPower per 40G port: Max: 31W & Typical: 25WF2B and B2F airflow SKUsUnique InnovationsECN for handling in-cast trafficOptimized multicast scheduling/replicationSupports flexible resource carvin

12、g (TCAM, Buffers)Latency Analytics with conditional SPAN, ERSPAN, IPFIX, Sampled Netflow26MB shared buffer per 12x 10GE ports or 3x 40GE portsUnmatched scale & featuresFCoE, NAT, Line rate SPAN/ERSPAN1024 way ECMP, 16K VLANs, 4K ACLLayer 2 Multi-Path (Fabric Path, TRILL)IPv4 & IPv6 Unicast & Multica

13、st Routing256K MAC/IP Host Tables with flexible carving32K LPM & 32K mroutesNexus 6004 Key Features尺寸及电源1RU; 48 个10G SFP+和4个40G QSFP+接口前进风或后进风交流或直流电源1+1 冗余电源创新特性采用基于条件的SPAN, ERSPAN进行延迟分析精准拥塞通告技术 (ECN) ，可以实现端到端的网络拥塞控制 优化组播规模及组播复制弹性的资源分配(TCAM, Buffers)大缓存设计：26MB shared buffer per 12x 10GE ports or 3x

14、40GE ports主要技术规格交换容量1.28Tbps线速三层转发，1us转发延迟256K MAC/ARP表，支持灵活分配Line rate SPAN/ERSPAN16K VLANs, 4K ACL32K LPM & 32K mroutesIPv4 & IPv6 Unicast & Multicast RoutingLayer 2 Multi-Path (Fabric Path, TRILL)FCoE, NAT, 1024 way ECMP, 支持24个FEX L2/L3模式Nexus 6001Cisco Nexus 5500Offering Choice and Flexibility f

15、or Varied DeploymentsNexus 5500 PlatformNexus 5548UP 1RU Switch (Unified Ports)Nexus 5596UP 2RU Switch (Unified Ports)Nexus 5596T 2RU Switch (10G BaseT)Nexus 5548 Layer 3 Daughter CardNexus 5596 Layer 3 Expansion ModuleNexus 5500 Unified Ports Expansion Module20 Million+2k/5k Ports Shipped125K+ Chas

16、sis Shipped 19,000+2k/5k CustomersNexus 5500 QSFP+ GEM1G10G40G201320122011Nexus 3048TP48-Port 1GT 4-Port SFP+Nexus 3016Q16-Port QSFPNexus 3132Q32-Port QSFP+Nexus 3064-X48-Port SFP 4-Port QSFP+ Integrated PHYNexus 3064-T48-Port 10G-BaseT 4-Port QSFP+ Nexus 3172PQ72-Port SFP+ Nexus 3172TQ72-Port 10GbT

17、Nexus 3064-32T32-Port 10GNexus 3000 SeriesN2224TP24 Port 100/1000M Host Interfaces2 x 10G UplinksN2232PP32 Port 1/10G FCoE Host Interfaces8 x 10G UplinksFET-10GCost Effective Fabric Extender Transceiver (5k/7k)N2248TP-E48 Port 100/1000M Host Interfaces4 x 10G Uplinks32MB Shared BufferB22: HP, Dell,

18、Fujitsu16 x 1/10G Host Interfaces 8 x 10G Uplinks (5k)N2232TM-E32 Port 1/10GBASE-T Host Interfaces8 x 10G Uplinks (Module)Lower Power and BER PHYN2232TM32 Port 1/10GBASE-T Host Interfaces8 x 10G Uplinks (Module)N2248TP48 Port 100/1000M Host Interfaces4 x 10G UplinksNexus 2200系列矩阵扩展器N2248PQ48 1/10G F

19、CoE Host Interfaces4 x QSFP 40G Uplinks思科数据中心接入层交换机定位Nexus 5000 & 6000Nexus 3000 & 3100Nexus 3500Switch-on-Chip (SoC)Multi-Stage FabricHigh Performance TradingHigh Performance ComputeWeb Services DCVirtualized DCUnified FabricUnified Fabric ReadyBuilt for VirtualizationHigh Performance ScaleLowest T

20、CO with Fabric Extender (FEX)High Density, High Performance, Low LatencyLow PowerOpen APIsProgrammableLowest LatencyDeepest VisibilityRich FeaturesProgrammableProgrammable DCNexus 2248TP-E大缓存机制Cisco Unified FabricMAPReduceNIF 1-4x10GEHIF 1-48dedicatedshared32MBIncastN2248TP-E Optimizations32MB 共享缓存避

21、免流量突发时出现丢包。增强的计数器跟踪Buffer的使用情况。N2248-TP-EModular SwitchFixed backplane=Fabric Extenders(Remote Line Cards)Cross-bar & Supervisor分布式模块化交换机10Gb Ethernet for the Backplane Physical Chassis Model Virtual Chassis ModelCisco FEX架构 简化管理，降低成本Nexus 7000/6000/5000 PlatformNexus 2000 Platform虚拟接入交换机（构成一个计算域）L3

22、L2接入层的虚拟化利用思科Nexus6000或Nexus7000作为“父交换机”，Nexus2000作为Fabric Extender，组成一个大型的虚拟接入层交换机整个虚拟接入层交换机本身“无环路”、“无阻塞”，很好的满足了部署服务器虚拟化的技术要求虚拟接入交换机. . .No Loop No STP利用vPC技术将两台设备虚拟为一台传统模式与vPC模式对比L2Non-vPCvPCVirtual Port ChannelPhysical TopologyLogical Topology利用vPC技术可以将两台设备虚拟化为一台设备，接入层设备可使用PortChannel方式上联，不必使用STP

23、协议可为虚拟机迁移、服务器集群提供可靠的二层网络环境提高了网络的可用带宽双上联的服务器可工作在Active-Active模式FabricPathVPC+FEXNexus5500Nexus7000VDC利用Fabric Path构建大二层网络Nexus5500支持16条链路的负载均衡（ DC1-5500-2(config)# fabricpath load-balance unicast ）利用ISIS协议构建Fabric Path路由表Conversational MAC地址学习，减小MAC地址表硬件设备已经具备支持TriLL的能力，通过软件升级即可支持Trill标准思科将推动把FbricPa

24、th的一些增强特性添加到Trill标准中WAN/Core边缘终结所有的主机二层协议：包括MAC学习, ARP, IGMP, LLDP, DHCP内部严格的故障隔离：Fabric内没有生成树、ARP广播、未知单播洪泛等等，象传统的路由网络一样可扩展VLAN Anywhere：任何服务器可在任意VLAN，VLAN可延展到全网Gateway Anywhere：任意位置都可做任意VLAN路由，同一VLAN网关一致Routing Anywhere：无论VLAN内还是VLAN之间都以类似三层路由的方式通信易于扩展、简化管理、自动部署DFA架构象三层路由网络一样的扩展和隔离性，实现统一的二层/三层转发GW

25、IP: 10.1.1.1GW MAC: 0011:2222:3333IS-IS：维护拓扑iBGP：通告主机路由FabricPath：交换机间转发新协议来自成熟技术灵活拓扑横向扩展高可用性动态接入带来优势提纲思科数据中心产品介绍典型组网方案成功案例Name NodeRuns a scheduler Job TrackerManages all data nodes, in memorySecondary Name Node Snapshot of meta data of HDFS clusterTypically all three JVM can run on single nodeData

26、 Node Task Tracker Receives Job Info from Job Tracker(Name Node)Map & Reducer Task Managed by Task TrackerConfigurable Ratio of Map & Reduce Task for various workload per Node/CPU/CoreData Locality - IF data not available where the map task is assigned, a missing block be copied over the networkHado

27、op Components and Operations25ToR FEX/switchData node 1Data node 2Data node 3Data node 4Data node 5ToR FEX/switchData node 6Data node 7Data node 8Data node 9Data node 10ToR FEX/switchData node 11Data node 12Data node 13Data node 14Data node 15Name NodeToR FEX/switchData node 1Data node 2Data node 3D

28、ata node 4Data node 5ToR FEX/switchData node 6Data node 7Data node 8Data node 9Data node 10ToR FEX/switchData node 11Data node 12Data node 13Data node 14Data node 15HDFS Architecture261SwitchName NodeFile metadata stored in fsimage.txt and in-memory only (!) map of blocks to data nodes/usr/sean/foo.

29、txt:blk_1,blk_2/usr/jacob/bar.txt:blk_3,blk_4Data node 1:blk_1Data node 2:blk_2, blk_3Data node 3:blk_3112223334444MapMapMapMapMapThe Data Ingest & ReplicationExternal ConnectivityEast West Traffic (Replication of data blocks)Map Phase Raw data Analyzed and converted to name/value pair.Workload tran

30、slate to multiple batches of Map task Reducer can start the reduce phase ONLY after the entire Map set is completeMostly a IO/compute functionHadoop Components and Operations27Hadoop Distributed File SystemUnstructured DataMapMapMapMapMapMapMapMapMapMapKey 1Key 1Key 1Key 1Key 1Key 1Key 1Key 2Key 1Ke

31、y 1Key 1Key 3Key 1Key 1Key 1Key 4ReduceShuffle PhaseReduceReduceResult/OutputReduceMapMapMapMapMapMapMapMapMapMapShuffle Phase - All name/value pair are sorted and grouped by their keys.Reducer is PULLING the data from the Mapper NodesHigh Network ActivityReduce Phase All values associates with a ke

32、y are process for results, three phasesCopy - Get intermediate result from each data node local diskMerge - To reduce the number of filesReduce MethodOutput Replication Phase - Reducer replicating result to multiple nodesHighest Network ActivityNetwork activities dependent on workload behaviorHadoop

33、 Components and Operations28Hadoop Distributed File SystemUnstructured DataMapMapMapMapMapMapMapMapMapMapKey 1Key 1Key 1Key 1Key 1Key 1Key 1Key 2Key 1Key 1Key 1Key 3Key 1Key 1Key 1Key 4ReduceShuffle PhaseReduceReduceResult/OutputReduceMapMapMapMapMapMap and Reduce Traffic29Many-to-Many Traffic Patte

34、rnMap 1Map 2Map NMap 3Reducer 1Reducer 2Reducer 3Reducer NHDFSOutput ReplicationNameNodeJobTrackerZooKeeperSmall Flows/Messaging(Admin Related, Heart-beats, Keep-alive, delay sensitive application messaging)Small Medium Incast(Hadoop Shuffle)Large Incast(Hadoop Replication)Network CharacteristicsThe

35、 relative impact of various network characteristics on Hadoop clusters* Not a scaled or measured data 接入层设计每个机柜的电力消耗不能超过13A，在每个机柜柜顶部署一台Nexus2248TP-E，服务器双网卡分别连接本机柜柜顶交换机及相邻机柜的柜顶交换机按照每个机柜13台服务器，加上带外管理接口，将占用39个端口Nexus 2248TP-ENexus 2248TP-E每个机柜13台服务器，每台服务器双网卡上联，也可以每3个机柜共用2台Nexus 2248TP-ENexus 2248TP-E千兆

36、接入方案一双挂方式，一个POD最多可支持24台N2248TP-EVPCNexus 7000VPCNexus 2248TP-EvPC4*10GNexus 6001POD1810GPOD2810GVPCInternet单挂方式，一个POD最多可支持24台N2248TP-E,PeerLink建议采用40G接口4*10G采用基于POD的设计思路，每个POD由2台Nexus6001+20台Nexus2248TP-E组成（最大支持24台N2000，阿里为16台），如果服务器多为单网卡接入，N2000可采用双挂方式提供更高的冗余能力，如果服务器采用双网卡聚合的方式接入，N2000可以采用单挂的方式，管理维护

37、更简单接入层采用4个万兆上行，可确保40个千兆接口线速转发（通常客户用不到40个千兆）在POD内是全线速转发的(一个POD可以支持200台服务器（按每机柜10台计算），绝大部分客户的Hadoop集群规模不会超过200台服务器）FEX虽然不具备本地转发能力，但从FEX-N6K-FEX的延迟4us小型客户，一个POD即可满足使用要求，大型客户，可以通过增加N7000作为核心交换机，基于POD进行横向扩展POD上联时可以考虑一定的收敛比，最大可支持16个10G捆绑或8个40G捆绑上联方案优势：POD内全线速；接入层大缓存；每POD只有2个管理节点；高可靠；便于横向扩展POD内端到端延迟All tes

38、ting are done in a port-pair configurationAll FEXs are connected to a single N5548 parent switchSpirent Test Center 3.6 and NX-OS 5.0(3)N1(1)CT: Cut-Through, SF: Store-ForwardNexus5548Nexus2000Sprient TEST Center千兆接入方案二VPCNexus 7000VPCNexus 3048/Catalyst 3850410G410GVPCInternet2台Nexus3048配置VPC可以支持服务

39、器双网卡聚合也可以采用2台Catalyst3850堆叠，同样支持服务器双网卡聚合Nexus3048更适合服务器单网卡上联的应用场景，整个DC架构扁平，没有POD限制，任意服务器之间均为全线速连接服务器双网卡连接的场景210GCatalyst2960XRNexus3048万兆接入方案一双挂方式，一个POD最多可支持24台N2248PQ-EVPCNexus 7700Nexus9500VPCNexus 2248PQvPC4*40GNexus 6004POD1840G/16*40GPOD2VPCInternet单挂方式，一个POD最多可支持12台N2232PP,PeerLink建议采用40G接口8*1

40、0G440GNexus 6001采用基于POD的设计思路，每个POD由2台Nexus6004+24台Nexus2248PQ组成。也可采用2台Nexus6001+12台Nexus2232构成一个POD。如果服务器多为单网卡接入，N2000可采用双挂方式提供更高的冗余能力，如果服务器采用双网卡聚合的方式接入，N2000可以采用单挂的方式，管理维护更简单强调N2248PQ的大缓存机制（32M共享缓存）FEX虽然不具备本地转发能力，但从FEX-N6K-FEX的延迟1us小型客户，一个POD即可满足使用要求，大型客户，可以通过增加N7000作为核心交换机，基于POD进行横向扩展POD上联时可以考虑一定的

41、收敛比，最大可支持16个40G捆绑上联Nexus 2232采用N2248PQ还是N2232按照覆盖36个机柜设计POD1包含24台N2248PQ，每三个机柜共用2台N2248PQ，可覆盖36个机柜POD2包含12台N2232，每个机柜柜顶部署1台N2232，需要3个POD可覆盖36个机柜方案一的收敛比为3:1，方案二的收敛比为4:1方案一的扩展性好于方案二方案一的布线成本高于方案二方案二有万兆电口的选择Nexus 2248PQvPC4*40GNexus 6004POD1POD28*10GNexus 6001Nexus 2232一个POD包含24台N2248PQ，按照三个机柜共用2台N2248P

42、Q设计，可覆盖36个机柜一个POD包含12台N2232，每个机柜柜顶部署1台N2232，覆盖36个机柜需要3个POD方案一：采用6004+N2248PQ方案二：采用6001+N2232万兆接入方案二VPCNexus 7700Nexus9500VPCNexus 3064Nexus3172840G440GVPCInternetTOR交换机具备本地转发能力整个DC架构扁平，没有POD限制，任意服务器之间均为全线速连接服务器双网卡连接的场景10GNexus 3064服务器单网卡连接的场景10GFeatureN3064N3172ArchitectureSwitch on ChipSwitch on Ch

43、ipRack Unit11Latency823nsec480nsecL2 ThroughputLine rateLine rateL3 ThroughputLine rateLine rate40GE support4610 GE support6472MAC Table128K288K (shared)IPv4 Unicast routes8K/16K (URPF disabled)16K(shared)IPv4 multicast routes8K32KACL support2K Ingress, 1K Egress4K Ingress, 1K EgressBuffer9 MB share

44、d 12MB shared 利用Spine-Leaf组网模型构建大型DCConfiguration Options:Layer 3 IP fabricLayer 3 all the way to accessLayer 2 extensibility can be provided by VXLANNexus 9500 can be EoRNexus 9300 can be ToR and MoRSpineLeafDC EdgeFabric Extenders2-Tier FabricRouting LeafsN9500/N7000 N9500/N7000 N9500/N7000 N9500/

45、N7000 N9300 N9300/N6000 N9300/N6000 N9300/N6000 N9300/N6000 N9300/N6000 N9300/N6000 N9500/N7000 N9500/N7000 提纲思科数据中心产品介绍典型组网方案成功案例6台Nexus2248Nexus5548POD 146台Nexus2248Nexus5548POD 1POD 内：Nexus5548 + TOR，用VPC+ 技术，消除STP，提供多路径均衡负载。每台N5K上行8个10GE，下行12个10GE，（考虑还有本地交互）每个POD6台TOR，上行4个10GE，下行48个GE，提供 288台服务器

46、的接入阿里巴巴Nexus7K * 2 VPCABTN支持4000台服务器配置14个POD，共支持4000台VPCOSPF ECMP(NG Fabric Ready)N6KN6KN6KN6KN6KN6KN6KN6KN6KN6KN6KN6KN2KN2KN2KN2KN2KN2KN2KN2KL3 GatwayLACPHost VPCDistri-SW(Spine)ACC-SW(Leaf)TOR(FEX)40GE面向NG Fabric的架构 (每个POD 768台服务器 1.28T上行)TOR层: - Nexus 2248TP-E, 采用4个10GE上行.（将来2248PQ，4个40G上行，全万兆接入）

47、POD SW层: 每台N6K带16个FEX, 上行16条40GE 链路做OSPF ECMP(将来NG Fabric）, 上下行收敛比1:1每个POD支持768 台服务器双网卡 A/A, 共组建8个POD, 支持 6144台服务器Core SW层:提供4台Nexus6004, 每台至少需要64个40GE端口新一代Alibaba模块内架构: N6k+N6k+N2kFEXCSWCSW腾讯数据中心网络设计原则As LARGE as it can be massively scalableAs FLAT as it can be unified low-subscriptionAs stupid as

48、is can be easily copiable & fast constructable & operableGeneral Guideline (1G Server)500M East-West traffic anywhere, which is 2:1 oversubscription rate5M South-North traffic per Server which has internet access140ms for 3-way-handshake is the bottom line for apps0.25kpps of ARP per Server, more than 2K Server within single VLAN could be dangerous45% average utilization rate for WAN/MAN LinksDC3.5.2DC3.5vDC3.5.3服务器容量 5,000 5,000 5,0004000+ * 10GE收敛比2:12:1 3:1虚拟化率NY, 4:1NY, 10:1 节点数量 5,000