下一代数据中心建设技术方案

1、下一代数据中心建设技术方案议题新一代数据中心的发展趋势新一代数据中心的网络架构数据中心建设方案Extranet接入1Extranet接入2安全控制安全控制安全控制安全控制Extranet1Extranet2主机生产业务B安全控制安全控制生产业务A前置3安全控制前置1安全控制前置2安全控制安全控制办公业务安全控制网管1网管2安全控制广域网区核心骨干网分行2分行1主网银备网银安全控制安全控制InternetInternet高速交换核心安全控制灾备中心安全控制传统架构的问题：硬件为中心的“竖井化”数据中心应用系统及服务通讯的应用服务器的基础架构网络基础架构基础设施(电力, 冷却,布线, 及物理安全)

2、存储的基础架构现有的基础架构缺乏灵活性相互难以配合的服务资源利用率低不一致的安全策略相互矛盾的优化方案不一致的业务永续/容灾计划物理资源无法复用重复人力物力投入僵化隔离的基础架构任何变动牵涉复杂效率 1525 - Gartner电力、制冷 40 - IDC部署新业务平均8周 - CIO Survey存储网络服务器?Applications计算资源网络资源存储资源用户界面企业应用系统业务需求虚拟机虚拟机虚拟机虚拟机智能业务调度机制构建云数据中心成为必然的发展方向将DC的IT资源看成云，以可计量的方式按需、保质的取用资源，屏蔽底层的复杂性，把精力投入到IT资源的应用上去优势：随需而动的业务灵活性自

3、动化的业务部署最佳的资源优化使用全局一致的服务可预测和有保证的业务服务质量高效节能、绿色环保 资 源 池 热备应用资源（数据库中间件）Host mobility: Hostsespecially virtual hostsmust be movable without interrupting existing connections or requiring address changes. Effective use of available bandwidth: A workload should not be limited by network bandwidth while usa

4、ble bandwidth exists along alternate paths. Self-conguration: Network elements must be able to forward trafc without manual conguration of forwarding tables. At scale, manual conguration makes management untenable.Scalability: The network should scale to accommodate the needs of modern data centers

5、without violating the preceding requirements. Scaling by hierarchically grouping smaller networks, e.g., groupingRespects Layering: The architecture must work with unmodied software stacks, e.g., operating systems and compute platforms, and higher layers must not need to understand details of the ar

6、chitectures implementation. Topology independent: The architecture must work with arbitrary topologies, e.g., Fat Tree, HyperX, or Jellysh. Restricting the network to a particular topology, e.g., Fat Tree, prevents network operators from considering alternate topologies that may offer better perform

7、ance, lower cost, or bothThe Modern Data Center Network MUST be optimized for Automation It must provide Reduction in Operational CostsIt must provide Provisioning and Operational Agility It must provide Improved Reliability It must be Open It must be focused on the Business Applications新一代数据中心网络需求新

8、一代数据中心需要灵活的基础架构以应用为中心的基础架构业务自动化自动化网络自动化管理业务自动化部署ComputeComputeStorageStorageServicesServicesL2,L3ProgrammableAutomationMonitoringAppsProvisioningAppsNetworkingAppsEnd-UserApps应用优化优化基础架构为应用优化基础架构应用可视化Congestion Management60%60%90%Dynamic Load BalancingDynamic Packet Prioritization资源融合融合ComputeCompute

9、StorageStorageServicesServicesSpine SwitchesL2,L3Leaf SwitchesFabric打破应用资源竖井资源整合化虚拟化议题新一代数据中心的发展趋势新一代数据中心的网络架构数据中心建设方案如何构建以应用为中心的基础架构资源融合融合ComputeComputeStorageStorageServicesServicesSpine SwitchesL2,L3Leaf SwitchesFabric打破应用资源竖井资源整合化虚拟化传统数据中心的业务分区竖井Extranet接入1Extranet接入2安全控制安全控制安全控制安全控制Extranet1Ext

10、ranet2主机生产业务B安全控制安全控制生产业务A前置3安全控制前置1安全控制前置2安全控制安全控制办公业务安全控制广域网区核心骨干网分行2分行1主网银备网银安全控制安全控制InternetInternet安全控制灾备中心安全控制资源利用率低物理资源无法复用重复人力物力投入缺乏灵活性僵化隔离的基础架构任何变动牵涉复杂高速交换核心网管1安全控制网管2安全控制10传统数据中心基本为南北向流量，流量模型相对固定由于虚拟化技术的采用，云计算数据中心将增加很多东西向流量（虚拟机迁移、虚拟机互访、集群技术的采用），流量模型更为复杂由于一台服务器承载了多个虚拟机，导致云计算数据中心的流量急剧增加，10G4

11、0G技术已经开始大量使用L3L2汇聚层接入层虚拟接入层汇聚层接入层虚拟接入层L2客户端服务器虚拟机虚拟机虚拟机虚拟机服务器服务器服务器服务器服务器服务器/服务器虚拟机新一代数据中心流量模型发生重大变化Change #5服务器服务器虚拟机虚拟机客户端服务器服务器服务器/服务器虚拟机服务器服务器分区1分区2对应传统分区的竖井化的网络架构VLAN 110L3L2VLAN 1120VLAN 2130Oracle RAC、HACMP等的集群化并行处理要求二层延展VMware、Hyper-V等虚拟环境需要自由灵活的虚机调度范围同部门服务器跨区延展（跨区的VLAN延展）FCoE等存储融合需要低延迟、高吞吐、

12、纯二层交换环境传统设计为何做不到真正的资源融合传统生成树传统跨机箱捆绑传统VXLAN传统VXLAN，传统二层多路径（TRILL）环路不稳定无负载均衡根本无法扩展不能横向扩展不稳定脑裂组播洪泛难于管理不支持三层路由VLAN/VXLAN无法通信WAN/Core边缘终结所有的主机二层协议：包括MAC学习, ARP, IGMP, LLDP, DHCP内部严格的故障隔离：Fabric内没有生成树、ARP广播、未知单播洪泛等等，象传统的路由网络一样可扩展VLAN Anywhere：任何服务器可在任意VLAN，VLAN可延展到全网Gateway Anywhere：任意位置都可做任意VLAN路由，同一VLAN

13、网关一致Routing Anywhere：无论VLAN内还是VLAN之间都以类似三层路由的方式通信需要全新一代的Fabric技术：NG Fabric象三层路由网络一样的扩展和隔离性，实现统一的二层/三层转发GW IP: GW MAC: 0011:2222:3333增强型的VXLANCisco NG Fabric灵活拓扑横向扩展高可用性动态接入带来优势动态、可横向扩展的拓扑NG Fabric提供灵活的拓扑推荐的拓扑形态节点小型化、分布扁平化（“胖POD”）动态平滑横向扩展：Pay as You Grow节点分布式、小型化：更细颗粒度的扩展高可用、高性能、低延迟简化管理如何构建以应用为中心的基础架

14、构应用优化优化基础架构为应用优化基础架构应用可视化Congestion Management60%60%90%Dynamic Load BalancingDynamic Packet Prioritization应用趋势：向小型化分布式计算发展存储资源计算资源不仅大数据计算，而且传统事务型应用也开始小型化、分布式处理，例如：VMware vSAN, SAP ERP on HANA，Mainframe / RISC / FTEnterprise x86 platformsOTTsHyperscale计算和存储节点本身的性能和可靠性对网络的依赖网络更多的参与到计算过程中网络流量模型的根本改变南北

15、东西Any Platform、Any Hypervisor、Any Application统一接入智能Fabric对各类业务转发需求是“可见的”高效跨网段通信、组播能力；安全管控内置完成；内置按需流量负载均衡；智能服务优化接入；流量智能转发，负载合理分布，消除流量热点，提高性能和可靠性高性能的智能统一Fabric为虚机、物理机、智能服务设备提供以流量最优化方式通信物理机VMware ESXMicrosoft Hyper-VRedhat KVMNVGREVXLANVLANVXLANIntelligent Fabric交换架构推荐采用40G主干虚拟化技术的采用，使服务器的网络吞吐量急剧增加服务器万

16、兆接入目前处于飞速发展阶段40G技术目前已经成熟，并开始商用部署，而且呈现出加速发展的势头考虑整个DC的技术先进性，建议交换架构骨干层采用40G互联综合布线要求采用40G布线系统（采用OM3、OM4光缆） 铜缆推荐采用6类、6A类、7类、7A类，支持1G、10G思科交换架构的40G BIDI技术更低成本，更简单的部署和管理传统40G新型40G75%端口成本75%布线成本应用的可视性：硬件辅助精确测量流量基于应用的遥测（Application Based Telemetry）输入监控目标比如：某应用的Web Server到App Server将收发两端的计数器联系在一起反馈结果Packets R

17、eceived on Leaf #5 sent from Leaf #1Path 12066Path 22963Path 32866Path 42506DifferencePath 12Path 20Path 33Path 40Packets sent from Leaf #1to Leaf #5Path 12068Path 22963Path 32869Path 42506PayloadIP应用标识Leaf地址M硬件设置标记应用的可视性：应用健康监测HYPERVISORHYPERVISORHYPERVISORHEALTH SCORELATENCYDROP COUNTVISIBILITYVMs

18、PhysicalApplication Delivery ControllerFirewall96%Microsecond(s)Packets Dropped525 73PayloadIPeVXLANiVTEPVNIDFlagsPolicyGroup应用标识的实现应用的可视性：基于健康状态的资源调整HEALTH SCORELATENCYDROP COUNTVISIBILITYVMsPhysicalApplication Delivery ControllerFirewall2352%Microsecond(s)Packets Dropped10350 96%525 168应用的可视性：应用级故

19、障定位和云租户视角管理如何构建以应用为中心的基础架构业务自动化自动化网络自动化管理业务自动化部署ComputeComputeStorageStorageServicesServicesL2,L3ProgrammableAutomationMonitoringAppsProvisioningAppsNetworkingAppsEnd-UserApps以应用为中心对各类软硬件资源实现优化调度 Access configuration, VLAN, VSAN, Security, and HardeningOperating System ConfigurationOS Type, Patch Le

20、vel, SettingsNetwork interface card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide names (WWNs), VSANs, and bandwidth constraints;and firmware revisionsUnique service ID, Application revisions,and Storage settingsApplication resources: Server, S

21、torage, Network Security, OSUplink port configuration, VLAN, VSAN, QoS, and EtherChannelsServer port configuration including LAN and SAN settingsNetwork interface card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide names (WWNs), VSANs, and bandw

22、idth constraints;and firmware revisionsUnique user ID (UUID), firmware revisions,and RAID controller settingsService profile assigned to server, chassis slot, or poolUplink port configuration, VLAN, VSAN, QoS, and EtherChannelsServer port configuration including LAN and SAN settingsNetwork interface

23、 card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide names (WWNs), VSANs, and bandwidth constraints;and firmware revisionsUnique user ID (UUID), firmware revisions,and RAID controller settingsService profile assigned to server, chassis slot, or

24、poolUplink port configuration, VLAN, VSAN, QoS, and EtherChannelsServer port configuration including LAN and SAN settingsNetwork interface card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide names (WWNs), VSANs, and bandwidth constraints;and fir

25、mware revisionsUnique user ID (UUID), firmware revisions,and RAID controller settingsService profile assigned to server, chassis slot, or poolUplink port configuration, VLAN, VSAN, QoS, and EtherChannelsServer port configuration including LAN and SAN settingsNetwork interface card (NIC) configuratio

26、n: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide names (WWNs), VSANs, and bandwidth constraints;and firmware revisionsUnique user ID (UUID), firmware revisions,and RAID controller settingsService profile assigned to server, chassis slot, or pool策略组合为Application Pro

27、file形成Application Network Profile，Server Profile，Storage Profile等在融合一体化的资源池内自动调配应用所需的各项资源234StorageSMEServerSMENetworkSME专项主题专家(SME)根据应用需求定义专项策略1SecuritySMEApp.SMEOSSMEApplication ProfileServer ProfileNetwork ProfileStorage ProfileSecurity Profile Access configuration, VLAN, VSAN, Security, and Hard

28、eningOperating System ConfigurationOS Type, Patch Level, SettingsNetwork interface card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide names (WWNs), VSANs, and bandwidth constraints;and firmware revisionsUnique service ID, Application revisions,

29、and Storage settingsApplication resources: Server, Storage, Network Security, OSUplink port configuration, VLAN, VSAN, QoS, and EtherChannelsServer port configuration including LAN and SAN settingsNetwork interface card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA con

30、figuration: worldwide names (WWNs), VSANs, and bandwidth constraints;and firmware revisionsUnique user ID (UUID), firmware revisions,and RAID controller settingsService profile assigned to server, chassis slot, or poolUplink port configuration, VLAN, VSAN, QoS, and EtherChannelsServer port configura

31、tion including LAN and SAN settingsNetwork interface card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide names (WWNs), VSANs, and bandwidth constraints;and firmware revisionsUnique user ID (UUID), firmware revisions,and RAID controller settingsS

32、ervice profile assigned to server, chassis slot, or poolUplink port configuration, VLAN, VSAN, QoS, and EtherChannelsServer port configuration including LAN and SAN settingsNetwork interface card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide na

33、mes (WWNs), VSANs, and bandwidth constraints;and firmware revisionsUnique user ID (UUID), firmware revisions,and RAID controller settingsService profile assigned to server, chassis slot, or pool策略组合为Application Profile形成Application Network Profile，Server Profile，Storage Profile等在融合一体化的资源池内自动调配应用所需的各

34、项资源234StorageSMEServerSMENetworkSME专项主题专家(SME)根据应用需求定义专项策略1SecuritySMEApp.SMEOSSMEApplication ProfileServer ProfileStorage ProfileSecurity Profile以应用为中心对各类软硬件资源实现优化调度Uplink port configuration, VLAN, VSAN, QoS, and EtherChannelsServer port configuration including LAN and SAN settingsNetwork interface

35、 card (NIC) configuration: MAC address,VLAN, and QoS settings;host bus adapter HBA configuration: worldwide names (WWNs), VSANs, and bandwidth constraints;and firmware revisionsUnique user ID (UUID), firmware revisions,and RAID controller settingsService profile assigned to server, chassis slot, or

36、poolNetwork Profile举例：Application Network Profile实现“应用定义的网络”通过控制器APIC（Application Policy Infrastructure Controller）ADCappdbf/wADCWEBAPICHYPERVISORHYPERVISORHYPERVISORConnectivity PolicySecurity PoliciesQoS Bandwidth Reservation AvailabilityStorage and computeApplication L4-L7 ServicesSLAQoSSecurityL

37、oadBalancingAPPLICATION NETWORK PROFILEExtensible Scripting Model新一代数据中心网络架构设计Spine（Nexus 9500）LeafNexus 93128Border LeafNexus 93128ECMP40GSpine（Nexus 9500）LeafNexus 9396LeafNexus 931281/10G1/10GInternet灾备中心外联区/安全服务区1/10GSpine（Nexus 9508）LeafNexus 93128Border LeafNexus 9396ECMP40G x n x 210GInternet

38、Spine（Nexus 9508）LeafNexus 9396LeafNexus 9396APIC控制器APIC控制器APICAPIC带外管理网10G10GRSLVSLVS添加APIC控制器演变化ACI组网模式网络服务设备的部署模式负载均衡设备资源池防火墙设备资源池动态添加安全服务可以根据业务需求，弹性的为物理或虚拟的应用添加防火墙、负载均衡等安全服务应用和安全服务可以由不同专业的人员负责，更易于划分管理界面APIC控制器是网络和服务的控制中心不管终端在任何位置，均可以灵活的添加网络服务Web ServerApp Tier AWeb ServerWeb ServerApp Tier BAppS

39、erver服务链“Security 5”策略重定向ApplicationAdminServiceAdminService GraphbeginendStage 1.Stage NProvidersinstinstFirewallinstinstLoad Balancer.Service Profile“Security 5” 服务链定义议题新一代数据中心的发展趋势新一代数据中心的网络架构数据中心建设方案数据中心现状InternetCatalyst 4506Catalyst 3750Catalyst 2960SGEGEGEGECatalyst 3750FCFC SAN网络改造第一阶段Intern

40、etNexus 9508Catalyst 2960S10G/40GGE/10GGEGEVPCCatalyst 3750Nexus93128TNexus9396ASA5585ASA5585Nexus 2248GEFCFC SAN网络改造第二阶段:向ACI组网模式演进InternetNexus 950810G/40GGE/10GNexus93128TNexus9396Service/Border LeafASA5585GE/10GNexus93128TASA Cluster（防火墙资源池）APIC控制器ClusterAPICAPICAPIC负载均衡资源池网络服务设备的部署模式负载均衡设备资源池防火

41、墙设备资源池动态添加安全服务可以根据业务需求，弹性的为物理或虚拟的应用添加防火墙、负载均衡等安全服务应用和安全服务可以由不同专业的人员负责，更易于划分管理界面APIC控制器是网络和服务的控制中心不管终端在任何位置，均可以灵活的添加网络服务Web ServerApp Tier AWeb ServerWeb ServerApp Tier BAppServer服务链“Security 5”策略重定向ApplicationAdminServiceAdminService GraphbeginendStage 1.Stage NProvidersinstinstFirewallinstinstLoad

42、Balancer.Service Profile“Security 5” 服务链定义利用集群技术实现防火墙性能线性提升ASA5585平台支持最多8台防火墙组成集群（9.2版本支持高达16台组成集群），共同工作好处：增强整体性能：8台cluster支持高达超过300G的性能投资保护：所有的节点都参与流量转发，实现按需扩展简化管理，cluster中的多台防火墙统一管理，统一配置，统一策略增加冗余度，各台防火墙之间存在备份机制，无中断升级单数据中心部署双活数据中心部署3/1/34/1/43/1/44/1/3N9K-13/1/34/1/43/1/44/1/3vPC 48vPC 41vPC 42Po 4

43、0Po 40N9K-20/60/70/80/90/60/70/80/9Po 48Data PathPort-ChannelControl Port-Channel建设多活数据中心数据中心1数据中心2SAN ExtensionsLAN Extensions数据中心互连访问优化哪些数据需要数据同步？对实时性要求如何？能否采用IP方式实现数据同步？如果一定要同步，可考虑两地三中心模式。不同数据中心之间是否有二层互通需求访问速度的优化通常采用CDN服务来实现多数据中心互联解决方案GEInternetNexus9396ASR1002XInternetXX机房XX机房ASR1002XGEGE数据中心1同城

44、数据中心ASR1002XASR1002XASR1002XOTVIpsec VPNIpsec VPN访问路径优化：智能DNSLayer 3 CoreIntranetISP AISP BAccessAggAccessVM= 00Default GW = DC ADC BVLAN A00KAL-AP Change IP000000GSSSNATSNATKAL-AP on VIPL2 Links (GE or 10GE)L3 Links (GE or 10GE)KAL-AP on VIPACEACE防火墙集群工作原理Inside NetworkOutside NetworkOwnerDirectorF

45、orwarderSYNClientServerSYN/ACK2: OwnerQuery4: SYN/ACK3:Owner location1: StateupdateSYN/ACKSYNAfter step 4, All remaining packets are forwarded directly to ownerASA Cluster利用vPC技术将两台设备虚拟为一台传统模式与vPC模式对比L2Non-vPCvPCVirtual Port ChannelPhysical TopologyLogical Topology利用vPC技术可以将两台设备虚拟化为一台设备，接入层设备可使用Port

46、Channel方式上联，不必使用STP协议可为虚拟机迁移、服务器集群提供可靠的二层网络环境提高了网络的可用带宽双上联的服务器可工作在Active-Active模式思科FEX架构简化管理，降低成本Modular SwitchFixed backplane=Fabric Extenders(Remote Line Cards)Cross-bar & Supervisor分布式模块化交换机10Gb Ethernet for the Backplane Physical Chassis Model Virtual Chassis ModelNexus 7000/6000/5000 PlatformNexus 2000 PlatformFEX技术实现接入层虚拟化虚