Dell EMC VxRail E series Deployment Manual Download Page 54

Page: 54 / 54

Dell EMC VxRail Multirack Deployment Guide

Support and feedback

Contacting Technical Support

Support Contact Information

Web:

http://www.dell.com/support

Telephone: USA: 1-800-945-3355

Feedback for this document

To provide feedback on the quality and usefulness of this publication, send an e-mail
to

[email protected]

Summary of Contents for VxRail E series

Page 1: ... using Network Virtualization Overlays NVO Abstract This document provides step by step deployment instructions for Dell EMC Networking OS10EE static VXLAN tunnels in an OSPF routed environment This provides the foundation that is needed for multirack VxRail host discovery and deployment in a modern data center February 2019 ...

Page 2: ...ally disclaims implied warranties of merchantability or fitness for a particular purpose Use copying and distribution of any software described in this publication requires an applicable software license 2019 Dell Inc or its subsidiaries All Rights Reserved Dell EMC Dell EMC and other trademarks are trademarks of Dell Inc or its subsidiaries Other trademarks may be trademarks of their respective o...

Page 3: ...vCenter connectivity 16 4 Switch configuration values and prerequisites 17 4 1 Pre planning VLANs IP address and switch values 17 4 1 1 VLANs and IP addresses 17 4 1 2 Switch settings 18 4 2 Switch prerequisites 18 4 2 1 Verify OS10EE version 18 4 2 2 Verify license installation 19 4 2 3 Factory default configuration 19 5 Configure switches 20 5 1 Configure general settings 20 5 2 Configure OSPF r...

Page 4: ...6 4 4 show vlt mac inconsistency 36 7 Perform initialization to create VxRail cluster 37 7 1 VxRail initialization 37 7 2 VxRail validation 38 8 VMware vSAN stretched clusters 39 8 1 Deploying VMware vSAN stretched clusters 40 8 1 1 Deploy the vSAN witness appliance 40 8 1 2 Configure vSAN witness appliance routing 41 8 1 3 Configure network interfaces for witness traffic 42 8 1 4 Configure fault ...

Page 5: ...5 Dell EMC VxRail Multirack Deployment Guide E 4 VxRail appliance software 52 F Technical resources 53 G Support and feedback 54 ...

Page 6: ...tensible LAN VXLAN VXLAN based solutions offer one of the most cost effective and straightforward paths to enable the routed underlay to forward L2 traffic between separate subnets Figure 1 shows Dell EMC Networking switches in Virtual Link Trunking VLT pairs connected over an OSPF enabled IP underlay using VXLAN tunnels This topology enables the creation of multiple virtual networks over one comm...

Page 7: ... guide does not provide step by step guidance on creating an Open Shortest Path First OSPF routed underlay or the specific steps for deploying VxRail post node discovery For detail instructions on creating a leaf spine underlay including alternative configurations see Dell EMC Networking Layer 3 Leaf Spine Deployment and Best Practices with OS10EE 1 2 Fabric Design Center FDC The Dell EMC Fabric D...

Page 8: ...witches and operating systems 1 4 Typographical conventions The CLI and GUI examples in this document use the following conventions Monospace Text CLI examples Underlined Monospace Text CLI examples that wrap the page Italic Monospace Text Variables in CLI examples Bold Monospace Text Commands entered at the CLI prompt or to highlight information in CLI output Bold text GUI fields and information ...

Page 9: ...l P series consists of high performance nodes that are optimized for heavy workloads such as databases Each appliance in the series has one node per 2 Rack Unit RU chassis The models within this series are the Dell EMC VxRail P570 hybrid and the Dell EMC VxRail P570F all flash These models are single or dual processor models based on the Dell EMC PowerEdge R740 rack server The example within this ...

Page 10: ...itch with up to 64 ports of multirate 100 GbE or up to 128 ports of 10 25 40 50 GbE ports using supported breakout cables The switch is a scalable L2 and L3 Ethernet switch with QoS and a full complement of standards based IPv4 and IPv6 features including OSPF and BGP routing support The example within this document uses two Z9264F ON switches as spine switches Dell EMC Networking Z9264F ON 2 4 De...

Page 11: ...four are tagged on each downstream interface Figure 8 shows the connections and the associated VLANs with the S5248F ON switches configured as a VLT pair As shown in Figure 8 the VxRail node iDRAC connects to the Dell EMC Networking S3048 ON which serves as the external management switch HA VxRail Node Spine Layer Mgmt Core Mgmt ToR S3048 ON iDRAC NIC2 NIC1 Leaf 1B S5248F ON Leaf 1A S5248F ON IP U...

Page 12: ... QSFP28 DD 200 GbE direct access cables DAC forming a VLT interconnect VLTi for a total throughput of 400 GbE Upstream connections to the spine switches are not shown but are configured using two QSFP28 100 GbE uplinks Stack ID Stack ID S5248F ON sfo01 leaf03 VxRail P node sfo01w01vxrail01 S5248F ON sfo01 leaf04 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16...

Page 13: ...e OSPF adjacency between the leaf switches and spines In addition with point to point interface mode there are no Type 2 link state advertisements LSAs Only type 1 LSAs are needed and this configuration model keeps the OSPF LSA database at a minimum size Note For detail instructions on creating a leaf spine underlay including considerations and alternative configurations see Dell EMC Networking La...

Page 14: ... QSFP28 DD 200 GbE cable pairs are used for VLTi Rack 2 Stack ID Stack ID Reset Stack ID Reset Stack ID Stack ID Stack ID S5248F ON sfo01 leaf03 S5248F ON sfo01 leaf04 S5248F ON sfo02 leaf03 S5248F ON sfo02 leaf04 Z9264F ON sfo spine01 Z9264F ON sfo spine02 Rack 1 Physical switch topology Note All switch configuration commands are provided in the file attachments See Section 1 4 for instructions o...

Page 15: ...equired VxRail VLANs which are shown as solid colored lines that are attached to virtual networks and that are associated with VTEP VXLANs at the leaf layer and shown as corresponding dashed lines Each leaf pair is configured with an identical VTEP and uses the same IP address on a loopback address Each VTEP is associated statically on the leaf switches The dashed lines represent the VXLAN tunnel ...

Page 16: ...shed lines represent different VXLAN tunnels connecting the different segments to the switch sfo edge01 Note For more information about how routing between tunnels is configured see Appendix A OSPF Area 0 Spine 1 Z9264 ON Spine 2 Z9264 ON sfo01w01 VxRail Nodes sfo01 leaf04 10 222 222 3 sfo01 leaf03 10 222 222 3 sfo01 leaf02 10 222 222 1 sfo01 leaf01 10 222 222 1 sfo01m01 Management cluster sfo edg...

Page 17: ...Node discovery vMotion Virtual machine migration vSAN Distributed storage traffic Guest VM networks One or more VLANs for VM data traffic Table 2 shows six VLANs VLAN IDs and IP network addresses planned for this deployment Two guest VM networks are defined as VLAN 10 and 20 VLANs and IP addresses VLAN ID Description Network CIDR Gateway VLAN Traffic 1631 External management 172 16 31 0 24 172 16 ...

Page 18: ... IP 10 255 2 4 31 10 255 2 5 31 10 255 2 6 31 10 255 2 7 31 Remote VTEPS 10 222 222 4 10 222 222 64 10 222 222 4 10 222 222 64 10 222 222 3 10 222 222 64 10 222 222 3 10 222 222 64 Note All switch configuration commands are provided in the file attachments See Section 1 4 for instructions on accessing the attachments 4 2 Switch prerequisites Before starting verify that all of the switches meet the...

Page 19: ...available for download on Dell Digital Locker See the OS10 Enterprise Edition User Guide for installation instructions 4 2 3 Factory default configuration The configuration commands begin with the Dell EMC Networking switches at their factory default settings Dell EMC Networking switches running the Dell EMC OS10 Enterprise Edition OS10EE can be reset to their default configuration using the follo...

Page 20: ...e At this point the configuration can continue from the console connection or an SSH session can be established using the management IP address The default username and password for OS10EE is admin admin It is suggested to change this to a more secure password 2 Set the hostname Unified Forwarding Table UFT mode RSTP priority value and the NTP server The Unified Forwarding Table UFT setting is a p...

Page 21: ...d add the interface to OSPF area 0 sfo01 leaf03 config interface ethernet 1 1 53 sfo01 leaf03 conf if eth1 1 53 description sfo spine01 sfo01 leaf03 conf if eth1 1 53 no shutdown sfo01 leaf03 conf if eth1 1 53 no switchport sfo01 leaf03 conf if eth1 1 53 mtu 9216 sfo01 leaf03 conf if eth1 1 53 ip address 192 168 1 9 31 sfo01 leaf03 conf if eth1 1 53 ip ospf network point to point sfo01 leaf03 conf...

Page 22: ...peer if there is a VLTi link failure sfo01 leaf03 config interface vlan 4000 sfo01 leaf03 conf if vl 4000 description L3 VLT peer sfo01 leaf03 conf if vl 4000 no shutdown sfo01 leaf03 conf if vl 4000 mtu 9216 sfo01 leaf03 conf if vl 4000 ip address 10 255 2 4 31 sfo01 leaf03 conf if vl 4000 ip ospf 1 area 0 0 0 0 sfo01 leaf03 conf if vl 4000 exit Note This guide uses the 2x 200 GbE QSFP DD ports t...

Page 23: ...4 sfo01 leaf03 conf vn vxlan vni remote vtep 10 222 222 64 sfo01 leaf03 conf vn vxlan vni exit sfo01 leaf03 conf vn 1631 exit 4 Enter the following commands to configure the VXLAN virtual network 3939 VxRail internal management traffic with a virtual network interface VNI and the appropriate remote VTEPs sfo01 leaf03 config virtual network 3939 sfo01 leaf03 conf vn 3939 description sfo01 w01 vxrai...

Page 24: ...vn 10 vxlan vni 10 sfo01 leaf03 conf vn vxlan vni remote vtep 10 222 222 4 sfo01 leaf03 conf vn vxlan vni remote vtep 10 222 222 64 sfo01 leaf03 conf vn vxlan vni exit sfo01 leaf03 conf vn 10 exit 8 Enter the following commands to configure the VXLAN virtual network 20 for the 2nd guest network with a virtual network interface VNI and the appropriate remote VTEPs sfo01 leaf03 config virtual networ...

Page 25: ...llowing commands to configure VLAN 1632 vMotion traffic and assign the VLAN to the VXLAN virtual network sfo01 leaf03 config interface vlan 1632 sfo01 leaf03 conf if vl 1632 description sfo01 w01 vmotion sfo01 leaf03 conf if vl 1632 virtual network 1632 sfo01 leaf03 conf if vl 1632 no shutdown sfo01 leaf03 conf if vl 1632 exit 4 Enter the following commands to configure VLAN 1633 vSAN traffic and ...

Page 26: ...1 1 1 flowcontrol receive on sfo01 leaf03 conf if eth1 1 1 flowcontrol transmit on sfo01 leaf03 conf if eth1 1 1 spanning tree port type edge sfo01 leaf03 conf if eth1 1 1 exit 2 Enter the following commands to configure the downstream interface for the second VxRail node Configure the interface as a trunk link Set the untagged VLAN to the VxRail external management VLAN 1631 Allow all remaining V...

Page 27: ...wnstream interface for the fourth VxRail node Configure the interface as a trunk link Set the untagged VLAN to the VxRail external management VLAN 1631 Allow all remaining VLANs on the trunk port Set MTU to the maximum size Flow control is enabled to receive and transmit matching the default VxRail ESXi NIC settings Spanning tree edge fast is enabled sfo01 leaf03 config interface ethernet 1 1 7 sf...

Page 28: ...d and tagged VLANs are listed In this example the four VxRail node interfaces show the six VLANs VLTi and upstream network facing ports as operational sfo01 leaf03 show interface status grep up Port Description Status Speed Duplex Mode Vlan Tagged Vlans Eth 1 1 1 sfo01w01 up 25G full T 1631 10 20 1632 1633 3939 Eth 1 1 3 sfo01w01 up 25G full T 1631 10 20 1632 1633 3939 Eth 1 1 5 sfo01w01 up 25G fu...

Page 29: ...7 f5 ee 40 vmnic0 ethernet1 1 50 sfo02 leaf04 ethernet1 1 50 54 bf 64 be f7 40 ethernet1 1 51 sfo02 leaf04 ethernet1 1 51 54 bf 64 be f7 40 ethernet1 1 52 sfo02 leaf04 ethernet1 1 52 54 bf 64 be f7 40 ethernet1 1 53 sfo spine01 ethernet1 1 17 20 04 0f 11 42 62 ethernet1 1 54 sfo spine02 ethernet1 1 17 20 04 0f 11 47 62 ethernet1 1 7 sfo01w01vxrail04 00 0a f7 f5 fa c0 vmnic0 6 1 4 show interface Th...

Page 30: ...spf neighbor Neighbor ID Pri State Dead Time Address Interface Area 10 0 2 6 1 FULL DR 00 00 39 10 255 2 5 vlan4000 0 0 0 0 10 0 1 1 1 FULL BDR 00 00 37 192 168 1 10 ethernet1 1 53 0 0 0 0 10 0 1 2 1 FULL BDR 00 00 39 192 168 2 10 ethernet1 1 54 0 0 0 0 6 2 2 show ip route ospf The show ip route ospf command shows the active routes for a network where the 10 222 222 0 24 subnet is used for VTEP en...

Page 31: ...10 3 21 53 01 O 192 168 2 0 31 via 192 168 2 10 eth1 1 54 110 3 21 52 55 O 192 168 2 2 31 via 192 168 2 10 eth1 1 54 110 3 21 52 55 O 192 168 2 12 31 via 192 168 2 10 eth1 1 54 110 2 21 52 55 O 192 168 2 14 31 via 192 168 2 10 eth1 1 54 110 2 21 52 55 O 192 168 2 252 31 via 192 168 2 10 eth1 1 54 110 3 21 52 55 6 2 3 show ip ospf topology The show ip ospf topology command verifies the structure of...

Page 32: ...two T Eth1 1 1 1 1 3 1 1 5 1 1 7 T Po1000 1631 Active sfo01 w01 vxrail_externa T Po1000 A Eth1 1 1 1 1 3 1 1 5 1 1 7 1632 Active sfo01 w01 vmotion T Eth1 1 1 1 1 3 1 1 5 1 1 7 T Po1000 1633 Active sfo01 w01 vsan T Eth1 1 1 1 1 3 1 1 5 1 1 7 T Po1000 3939 Active sfo01 w01 vxrail_interna T Eth1 1 1 1 1 3 1 1 5 1 1 7 T Po1000 4000 Active L3 VLT peer T Po1000 4094 Active T Po1000 6 3 2 show nve remote...

Page 33: ...source interface is being used to establish the VTEP endpoint and what remote VTEPs the virtual network terminates at sfo01 leaf03 show virtual network 1631 Codes DP MAC learn Dataplane CP MAC learn Controlplane UUD Unknown Unicast Drop Virtual Network 1631 Description sfo01 w01 vxrail_external_management Members Untagged ethernet1 1 1 ethernet1 1 3 ethernet1 1 5 ethernet1 1 7 VLAN 1631 port chann...

Page 34: ...4 c7 d4 c0 Role priority 32768 VLT MAC address 54 bf 64 c7 d4 c0 IP address fda5 74c8 b79e 1 1 Delay Restore timer 90 seconds Peer Routing Enabled Peer Routing Timeout timer 0 seconds VLTi Link Status port channel1000 up VLT System MAC Address Status IP Address Version 2 54 bf 64 c7 e5 40 up fda5 74c8 b79e 1 2 2 0 6 4 2 show vlt all backup link The show vlt all backup link command displays that VL...

Page 35: ...o mismatch Peer routing mismatch No mismatch VLAN mismatch No mismatch VLT VLAN mismatch No mismatch VLT Virtual Network Mismatch Virtual Network Name Mismatch No mismatch Virtual Network VLTi VLAN Mismatch No mismatch Virtual Network Mode Mismatch No mismatch Virtual Network Tagged Interfaces Mismatch No mismatch Virtual Network Untagged Interfaces Mismatch No mismatch Virtual Network VNI Mismatc...

Page 36: ...addresses that are learned between VLT peers across spanned VLANs or virtual networks Note Verify that the No inconsistencies found status displays sfo01 leaf03 show vlt mac inconsistency virtual network Inconsistency check for Virtual Network based MAC Fetching VN MACs from unit 2 Fetching VN MACs from unit 1 Comparing MACs of VLT Peers No inconsistencies found ...

Page 37: ...ts of the VxRail nodes and the switch ports 3 Power on the four primary E series nodes in Rack 1 to form the initial VxRail cluster Note Do not turn on the nodes in Rack 2 4 To access the VxRail external management on VLAN 1631 connect a workstation or laptop that is configured for VxRail 5 Using a web browser go to the VxRail default IP address to begin the VxRail initialization process 6 Complet...

Page 38: ...hree of the four nodes were used however all four can be selected in a typical cluster expansion scenario VxRail manager cluster expansion 7 2 VxRail validation Once the deployment and expansion of the initial cluster are completed use the VxRail Manager to verify that all eight hosts are in the single cluster VxRail Manager logical view ...

Page 39: ...contains the vSAN witness appliance whose purpose is to host the witness components of virtual machine objects Rack 1 and Rack 2 have already been configured using VXLANs stretched the Layer 2 L2 domains between the two racks Traffic to the vSAN witness appliance is done through L3 routing between the two data centers Note For more information about how routing between tunnels is configured see Ap...

Page 40: ...enter to an existing VMware cluster lax01 m01 mgmt01 vSAN witness appliance Table 4 shows the values that are used during the vSAN witness appliance deployment vSAN witness appliance configuration details VLAN Network label IP address Purpose 1731 Management Network 172 17 31 201 24 ESXi management VMkernel 1733 witnessPg 172 17 33 201 24 vSAN VMkernel Once deployed sfo03w01vsan01 is then added to...

Page 41: ...4 using the next hop address for second data center vSAN network 172 17 33 0 24 root sfo03w01vsanw01 esxcli network ip route ipv4 add n 172 16 31 0 24 g 172 17 33 253 3 Use the vmkping command to verify reachability from the vSAN Witness Appliance vmk1 interface to the management IP address for the first VxRail node root sfo03w01vsanw01 vmkping I vmk1 172 16 31 101 PING 172 16 31 101 172 16 31 101...

Page 42: ... 255 255 255 0 172 16 33 255 STATIC 0 0 0 0 vmk4 172 16 32 101 255 255 255 0 172 16 32 255 STATIC 0 0 0 0 vmk1 169 254 0 2 255 255 255 0 169 254 0 255 STATIC 0 0 0 0 3 Use the esxcli vsan network ip add command to configure the management VMkernel network adapter to support witness traffic root sfo01w01vxrail01 esxcli vsan network ip add i vmk2 T witness 4 Use the esxcli vsan network list command ...

Page 43: ...in Rack ID Node membership Preferred 1 sfo01w01vxrail01 sfo01w01vxrail02 sfo01w01vxrail03 sfo01w01vxrail04 Secondary 2 sfo01w01vxrail05 sfo01w01vxrail06 sfo01w01vxrail07 sfo01w01vxrail08 Table 6 shows the vSAN witness appliance disk settings that are selected during the fault domain creation wizard These settings are specific to the vSAN witness appliance Witness appliance settings Setting Value L...

Page 44: ...ect the VxRail Manager virtual machine object 6 Click View Placement Details Figure 21 shows the VxRail Manager virtual machines virtual objects This view shows virtual object Hard disk 1 using the default RAID 1 mirror vSAN storage policy Components are mirrored between VxRail node sfo01w01vxrail01 in the Preferred fault domain and sfo01w01vxrail06 in the Secondary fault domain and the witness co...

Page 45: ...53 sfo01w01vxrail01 IP 172 16 31 101 24 GW 172 16 31 253 VLAN 1611 IP 172 16 11 253 24 VLAN 1631 IP 172 16 31 253 24 Routing between L2 VXLAN virtual networks The vCenter server sfo01w01vc01 sfo01 rainpole local has the IP address 172 16 11 62 24 and the first VxRail node sfo01w01vxrail01 has the IP address 172 16 31 101 24 In the current implementation of a static L2 VXLAN routing in and out of t...

Page 46: ...ng a port channel that carries the VLAN that requires routing The show vlan command shows the VLAN membership as well as the virtual network association sfo edge01 show vlan except default Codes Default VLAN M Management VLAN R Remote Port Mirroring VLANs Attached to Virtual Network Q A Access Untagged T Tagged NUM Status Description Q Ports 10 Active sfo01 w01 guest_vm_network_a T Po10 20 Active ...

Page 47: ...mation from Layer 3 and above VXLAN reduces the MTU to 1450 bytes through its default configuration This is because VXLAN adds either 50 or 54 bytes of overhead as part of identifier information Figure 23 shows a standard 54 byte VXLAN packet with an 802 1Q tag Original L2 Frame Outer MAC Inner IP UDP VXLAN SMAC 802 1Q Etype CRC New DMAC 14B 20B 8B 8B 18B 4B Payload SMAC 802 1Q Etype DMAC Payload ...

Page 48: ...me FQDN sfo01w01vc01 sfo01 rainpole local vCenter Server admin username password vxrail admin sfo01 rainpole local vCenter Server SSO domain rainpole local Data center name sfo01 w01dc New cluster name sfo01 w01 vxrail01 Platform Services Controller PSC Hostname sfo01psc0101 sfo01 rainpole local VxRail Manager VxRail Manager hostname sfo01w01vxrail mgr VxRail IP address 172 16 31 71 Subnet mask 25...

Page 49: ...e 1 1 51 QSFP28 DD QSFP28 DD 2x100GBASE 2CR4 0 5M CNFCK0077K0017 true 1 1 52 QSFP28 DD QSFP28 DD 2x100GBASE 2CR4 0 5M CNFCK0077K0017 true 1 1 53 QSFP28 QSFP28 100GBASE SR4 CN01360385M001F true 1 1 54 QSFP28 QSFP28 100GBASE SR4 CN01360364N000M true 2 The show port group command is used to identify which port group the SFP 1GbE Base T adapter belongs to Here port group 1 1 8 contains physical interf...

Page 50: ...af03 conf if eth1 1 32 1 switchport access vlan 1631 5 Use show interface to validate that the interface is operating at the correct speed sfo02 leaf03 show interface status grep up Port Description Status Speed Duplex Mode Vlan Tagged Vlans Eth 1 1 1 sfo01w01 up 25G full T 1631 10 20 1632 1633 3939 Eth 1 1 3 sfo01w01 up 25G full T 1631 10 20 1632 1633 3939 Eth 1 1 5 sfo01w01 up 25G full T 1631 10...

Page 51: ...the information that is provided below VxRail E560 node components Qty per node Item Firmware version 2 Intel Xeon Gold 6136 CPU 3 00GHz 12 cores 12 16GB DDR4 DIMMs 192GB total 2 800GB SAS SSD 8 1 2TB SAS HDD EF03 1 Dell HBA330 Storage Controller 15 17 09 06 1 Boot Optimized Storage Solution BOSS Controller w 2x240GB SATA SSDs 2 5 13 3016 1 Broadcom 57414 NDC 2x25GbE SFP28 ports 20 08 04 04 BIOS 1...

Page 52: ...RAC with Lifecycle Controller 3 21 23 22 E 4 VxRail appliance software Chapter 10 of this deployment guide was developed using VxRail appliance software 4 7 0 The software consists of the component versions that are provided below VxRail appliance software component versions Item Version Dell PTAgent 1 8 1 21 VMware ESXi 6 7 0 build 10764712 VxRail Manager 4 7 001 11073036 VMware vCenter Server Ap...

Page 53: ...ion Users Guide 10 4 2 0 Edition Dell EMC Networking Layer 3 Leaf Spine Deployment and Best Practices with OS10EE Dell EMC VxRail Network Guide VxRail Appliance 4 7 Administration Guide VxRail Planning Guide for Virtual SAN Stretched Cluster VMware Validated Design for Software Defined Data Center 4 3 Release Notes ...

Page 54: ...ck Contacting Technical Support Support Contact Information Web http www dell com support Telephone USA 1 800 945 3355 Feedback for this document To provide feedback on the quality and usefulness of this publication send an e mail to Dell_Networking_Solutions Dell com ...

Search results

Summary of Contents for VxRail E series

Reviews:

Related manuals for VxRail E series

Brands by name

Popular brands

Dell EMC VxRail E series, Deployment Manual

Search results

Summary of Contents for VxRail E series

Reviews:

Related manuals for VxRail E series

Brands by name

Popular brands