Monday, December 28, 2015

VMware NSX useful resources

I'm trying to deep dive into VMware Network Virtualization (NSX) and I have decided to collect all useful resources I will find during my journey.

There are two NSX flavors. NSX-v and NSX-T. It is good to read NSX-v vs NSX-T: Comprehensive Comparison to understand differences.

Fundamentals
NSX-V Design
NSX-T Design
NSX-V Operations
NSX-T Operations
NSX Automation
NSX in Home Lab
NSX Advanced
NSX Security
NSX Dynamic Routing


Networking 


NSX Tutorial


Other lists of resources
  • Rene Van Den Bedem (@vcdx133) :  NSX Link-O-Rama - great list of resources gathered by Rene
Tools
  • ARKIN - network visibility and analytics
This list will be continuously updated.
If you know any other useful NSX resource don't hesitate to write a comment with link.

Posted by at 1 comment:
Labels:

Thursday, December 17, 2015

End to End QoS solution for Vmware vSphere with NSX on top of Cisco UCS

I'm engaged on a private cloud project where end to end network QoS is required to achieve some guarantees for particular network traffics.  These traffics are
  • FCoE Storage
  • vSphere Management
  • vSphere vMotion
  • VM production
  • VM guest OS agent based backup <== this is the most complex requirement in context of QoS
Compute and Network Infrastructure is based on
  • CISCO UCS
  • CISCO Nexus 7k and 
  • VMware NSX. 
More specifically following hardware components has to be used:
  • CISCO UCS Mini Chassis 5108 with Fabric Interconnects 6324 
  • CISCO UCS servers B200 M4 with virtual interface card VIC1340 (2x10Gb ports - each port connected to different fabric interconnect)
  • CISCO Nexus 7k
Customer is also planning to use NSX security (micro segmentation) and vRealize Automation for automated VM provisioning.

So now we understand the overall concept and we can consider how to achieve end to end network QoS to differentiate required network traffics. 

Generally we can leverage Layer 2 QoS 802.1p (aka CoS - Class of Service ) or Layer 3 QoS (aka DSCP - Differentiated Services Code Point). However, to achieve end to end QoS on Cisco UCS we have to use CoS because it is the only QoS method available inside Cisco UCS blade system to guarantee bandwidth in shared 10Gb NIC (better to say CNA) ports.

The most important design decision point is where we will do CoS marking to differentiate required network traffics. Following two options are generally available:
  1. CoS marking only in Cisco USC (hardware based marking) 
  2. CoS marking on vSphere DVS portgroups (software based marking)
Lets deep dive into available options.

OPTION 1 - UCS Hardware CoS marking

Option 1 is depicted on figure below. Please, click on figure to understand where CoS marking and bandwidth management is done.


Following bullets describes key ideas of option 1:
  • Management and vMotion traffics are consolidated on the same pair of 10Gb adapter ports (NIC-A1 and NIC-B1) together with FCoE traffic. Active / Standby teaming is used for vSphere Management and vMotion portgroups where each traffic is by default active on different UCS fabric.
  • VTEP and Backup traffics are consolidated on the same pair of 10Gb adapter ports (NIC-A2 and NIC-B2). Active / Standby teaming is used for NSX VTEP and backup portgroup.  Each traffic is by default active on different UCS fabric. 
  • Multiple VTEPs and Active / Active teaming for backup portgroup can be considered for higher network performance if necessary.
  • All vmkernel interfaces should be configured consistently across all ESXi hosts to use the same ethernet fabric in non-degraded state and achieve optimal east-west traffic.
Option 1 implications:
  • Two Virtual Machine's vNICs has to be used because one will be used for production traffic (VXLAN) and second one for backup traffic (VLAN backed portgroup with CoS marking).
OPTION 2 - vSphere DVS CoS marking:

Option 2 is depicted on figure below. Please, click on figure to understand where CoS marking and bandwidth management is done.

Following bullets describes key ideas of Option 2 and difference against Option 1:
  • Management and vMotion traffics are also consolidated on the same pair of 10Gb adapter ports (NIC-A1 and NIC-B1) together with FCoE traffic. The difference against Option 1 is usage of single CISCO vNIC and CoS marking in DVS portgroups. Active / Standby teaming is used for vSphere Management and vMotion portgroups where each traffic is by default active on different UCS fabric.
  • VTEP and Backup traffics are consolidated on the same pair of 10Gb adapter ports (NIC-A2 and NIC-B2). Active / Standby teaming is used for NSX VTEP and backup portgroup.  The difference against Option 1 is usage of single CISCO vNIC and CoS marking in DVS portgroups. Each traffic (VXLAN, Backup) is by default active on different UCS fabric. 
  • Multiple VTEPs and Active / Active teaming for backup portgroup can be considered for higher network performance if necessary.
  • All vmkernel interfaces should be configured consistently across all ESXi hosts to use the same ethernet fabric in non-degraded state and achieve optimal east-west traffic.
  • FCoE traffic is marked automatically by UCS for any vHBA. This is the only hardware based CoS marking.
Option 2 implications:
  • Two Virtual Machine's vNICs has to be used because one will be used for production traffic (VXLAN) and second one for backup traffic (VLAN backed portgroup with CoS marking).

Both options above requires two vNICs per VM. It is introducing several challenges some of them listed below:
  1. More IP addresses required
  2. Default IP gateway is used for production network, therefore, the backup network cannot be routed without static routes inside guest OS. 
Do we have any possibility to achieve QoS differentiation between VM traffic and In-Guest backup traffic with single vNIC per VM? 

We can little bit enhance Option 2. Enhanced solution is depicted in the figure below. Please, click on figure to understand where CoS marking and bandwidth management is done.


So where is the enhancement? We can leverage enhanced conditional CoS marking in each DVS portgroup used as NSX virtual wire (aka NSX Logical Switch or VXLAN). If IP traffic is targeted to the backup server we will mark it as CoS 4 (backup) else we will mark it a CoS 0 (VM traffic). 

You can argue VXLAN is L2 over L3 and thus L2 traffic where we did a conditional CoS marking will be encapsulated into L3 traffic (UDP) in VTEP interface and we will lose CoS tag. However, that's not the case. VXLAN protocol is designed to keep L2 CoS tags by copying inner Ethernet header into outer Ethernet header. Therefore virtual overlay CoS tag is kept even in physical network underlay and it can be leveraged in Cisco UCS bandwidth management (aka DCB ETS - Enhanced Transmission Selection) to guarantee bandwidth for particular CoS traffics.

The enhanced Option 2 seems to me as the best design decision for my particular use case and specific requirements.

I hope it makes sense and someone else find it useful. However, I share it with IT community for broader review. Any comments or the thoughts are very welcome.

Posted by at 3 comments:
Labels: , , , , , ,

Monday, November 16, 2015

Creating a Capacity & Performance Management Dashboard in vRealize Operations 6.x

I'm long time proponent of performance SLAs in modern virtual datacenters. Performance SLAs is nothing else than mutual agreement between service provider and service consumer. Agreement describes what performance of particular resource consumer can expect and provider should guarantee. The performance SLA is important mainly on shared resources. On dedicated resources consumer knows exactly what to expect from performance point of view. In modern virtualized datacenters is almost everything shared therefore performance SLAs are a must and all service consumers should require it.

The most important shared resources on virtualized infrastructures having significant impact on application performance are CPU and Disk. The rest infrastructure resources - Memory and Network - are important as well but CPU and Disk performance was typical final root cause of any performance troubleshootings I did over several years. In VMware vSphere we can typically identify CPU Contention by CPU %RDY metric and Storage Contention based on disk response time of normalized I/O's. We can identify such issues during troubleshooting when infrastructure consumers are complaining about application performance. We call it reactive approach.  But more mature approach is to identify potential performance issues before application is affected.  We call it proactive approach. And that's where performance SLA's and threshold monitoring come in to play.

Infrastructure performance SLA can looks like

  • CPU RDY is below 3% (notification threshold 2%)
  • If # of vDisk IOPS < 1000 then vDISK Response Time is below 10ms (notification threshold 7ms)

Simple right? These two bullets above should be clearly articulated, explained and agreed between infrastructure service provider and infrastructure consumer building and providing application services on top of infrastructure.

So now how to monitor these performance metrics? I have just found Sunny Dua and Iwan Rahabok blog post covering this topic and step by step problem solution with vRealize Operations 6.x. Sunny and Iwan prepared and shared with community customized vROps supermetrics, views and dashboards for performance capacity planning.  To be honest I do not have big experience with vROps so far but it seems to me as very helpful tool for anybody using vRealize Operations as monitoring platform.

Let's try to build and provide mature IT with clearly articulated SLA's and with agreed expectations between service providers and service consumers.
Posted by at No comments:
Labels: , ,

Friday, November 06, 2015

VMware Tools 10 and "shared productLocker"

VMware tools (aka VM tools, vmtools) were always distributed together with ESXi image however it changed with VMware Tools 10. VMware is now shipping VM tools also outside of the vSphere releases. For more information look at this blog post.

Where can I get VMware Tools?

Option 1/ VMware Tools 10 can be downloaded from my.vmware.com. More specifically from this direct URL. Please be aware, that you must be logged in to my.vmware.com before direct link works.

Option 2/ VMware Tools can be also downloaded from here without any login credentials required. The latest version (10.0.6 at the moment of writing this blog post) is available here. Benefit of option (2) is that there are binaries which can be use directly within guest operating systems.

Option 3/ Open-vm-tools. This option is available for linux based and FreeBSD operating systems. You can read more about it here , in SourceForge or in GitHub. Optimally the open-vm-tools should be available through standard package manager of your unix-like operating system.

It is worth to mention that
  • VMware Tools 10 are backward compatible and are independent on ESXi version.
  • You can share VMware Tools 10 packages with application/OS owners and they can update VMware Tools by them selves during OS update procedure. But even your OS owners will do VMtools update by them selves it is still worth to have VMware tools available in ESXi to have ability of VMtools comparison from vSphere point of view.
VMtools versions

VMtools reports version as a number. For example version number 9536 is version 9.10.0. You can map VMtools version number to human readable version by leveraging this file.

Updates

Ok, so what? If you update your VMware Tools in old way (together with ESXi image) you - or VMware Update Manager - have to upload VMware tools to each ESXi host with following impacts
  1. It takes significantly more time especially in bigger environments.
  2. You can potentially end-up with different VM tools version in different ESXi hosts in your datacenter. It can be reported as outdated VM tools after vMotion of particular VM across different ESXi hosts.
The thing is that VMware Tools 10 and above doesn't need to be updated automatically with ESXi update on each ESXi host. You can update ESXi hosts without VMware Tools and later update VMware tools bundle just on single shared place - in shared productLocker location.

So what actually the "productLocker" is? The "productLocker" is essentially VMware Tools directory. This directory is on each ESXi host by default however it can be reconfigured and pointed to directory on shared datastore. Such configuration is called "shared productLocker" and it enables us to do centralize update of VM tools. It is worth to mention that all your ESXi hosts must be reconfigured to use this shared location.

Reconfiguration has to be done via ESXi host advanced configuration option UserVars.ProductLockerLocation. So it has to be changed manually on each host, you can change it automatically via custom PowerCLI script or if you have Enterprise Plus Edition you can leverage Host Profiles. The last option works for me perfectly.

Below is screenshot showing /productLocker directory structure and content on ESXi 6 host ...

/productLocker directory structure and content 
If you use central location for VMware Tools then you don't need update ESXi hosts with full ESXi image but only with ESXi image without VMware Tools. See example of different profiles in ESXi 6 Update 2 image below.
[root@esx01:~] esxcli software sources profile list -d /vmfs/volumes/NFS-SYNOLOGY-SATA/ISO/update-from-esxi6.0-6.0_update02.zip
Name                              Vendor        Acceptance Level
--------------------------------  ------------  ----------------
ESXi-6.0.0-20160301001s-no-tools  VMware, Inc.  PartnerSupported
ESXi-6.0.0-20160302001-standard   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20160301001s-standard  VMware, Inc.  PartnerSupported
ESXi-6.0.0-20160302001-no-tools   VMware, Inc.  PartnerSupported
Profile names with postfix no-tools can be used for ESXi update without updating VMware Tools to each ESXi host. For further details how to update ESXi hosts with particular profile see my other post - How to update ESXi via CLI.

Current ESXi host product locker location can be displayed by esxcli command
esxcli system settings advanced list -o /UserVars/ProductLockerLocation 
and the output should looks like ...
[root@esx01:~] esxcli system settings advanced list -o /UserVars/ProductLockerLocation   
Path: /UserVars/ProductLockerLocation   
   Type: string 
   Int Value: 0
   Default Int Value: 0
   Min Value: 0   
   Max Value: 0   
   String Value: /locker/packages/6.0.0   
   Default String Value: /locker/packages/6.0.0   
   Valid Characters: *   
   Description: Path to VMware Tools repository
To change location you can use following esxcli command
esxcli system settings advanced set -o /UserVars/ProductLockerLocation --string-value "/vmfs/volumes/NFS-SYNOLOGY-SATA/VMtools/latest" 
And you can verify that setting was changed ...
[root@esx02:~] esxcli system settings advanced list -o /UserVars/ProductLockerLocation
   Path: /UserVars/ProductLockerLocation
   Type: string
   Int Value: 0
   Default Int Value: 0
   Min Value: 0
   Max Value: 0
   String Value: /vmfs/volumes/NFS-SYNOLOGY-SATA/VMtools/latest   
   Default String Value: /locker/packages/6.0.0   
   Valid Characters: *   
   Description: Path to VMware Tools repository
ESXi host has to be rebooted to activate new Product Locker Location.

Hope this helps other folks in VMware community to simplify operations with VMware Tools.

UPDATE 2018-02-05: I have just been told about very nice PowerCLI command-lets allowing to manage VMtools.  Command-let Update-VMToolsImageLocation updates the link /productLocker in ESXi host directly to avoid host reboot and command-let Invoke-VMToolsUpgradeInVMs in the combination with shared productLocker is a very nice way how to automatically update VMtools.
All VMtools management command-lets are available on GitHub here
https://github.com/vmware/PowerCLI-Example-Scripts/blob/master/Modules/VMToolsManagement/VMToolsManagement.psm1

UPDATE 2019-04-04: New VMware blog post about this topic has been published in January 2019. The blog post is here Configuring a VMware Tools Repository in vSphere 6.7U1.
In comments is the link to PowerCLI code leveraging vSphere API to change the productLocker location. The link to the code is here. The code is clear and simple ...

 $esxName = 'MyEsx'  
 $dsName = 'MyDS'  
 $dsFolder = 'ToolsRepo'  
 $esx = Get-VMHost -Name $esxName  
 $ds = Get-Datastore -Name $dsName  
 $oldLocation = $esx.ExtensionData.QueryProductLockerLocation()  
 $location = "/$($ds.ExtensionData.Info.Url.TrimStart('ds:/'))$dsFolder"  
 $esx.ExtensionData.UpdateProductLockerLocation($location)  
 Write-Host "Tools repository moved from"  
 Write-Host $oldLocation  
 Write-Host "to"  
 Write-Host $location

References


Posted by at No comments:
Labels: , , ,

Wednesday, October 07, 2015

How to restore deleted vmdk from VMFS5

Yesterday I have got an E-mail from somebody asking me how to restore deleted vmdk from VMFS5. They deleted VM but realised there are very important data.

Typical answer would be - "Restore from backup" - however they wrote that they don't have backup.

Fortunately, I have never had a need to restore deleted vmdk so I was starting to do some quick research (aka googling :-) )

I found VMware KB 1015413 with following statement ...
"VMware does not offer data recovery services. This article provides contact information for data recovery service providers.
Note: VMware does not endorse or recommend any particular third-party utility, nor is this list meant to be exhaustive."
I was sitting in the VMware office so I have asked colleagues if they have any practical experience with undeleting vmdk from VMFS.  One colleague of mine suggested utility "VMFS Recovery" from DiskInternals.com. He had positive experience with this particular tool. His suggestion was to use trial version which should help to identify if recovery is possible and buy full version for recovery.

Warning: Use any third party tool for recovery on your own risk.

I absolutely understand that if you lose your important data you would like to try anything to recover it back however here are my general thoughts:

  • Clone or snapshot your raw disk (LUN) with VMFS before any recovery (use storage array capabilities, third party imaging tool or *nix dd command) 
  • If your data are very valuable for you consider engagement of data recovery expert services
  • When  you do recovery by your self I wish you good luck.
If you have some other experience with this topic please share it with community in comments. 
Posted by at 1 comment:
Labels: , , , , ,

Thursday, October 01, 2015

VMware VM Template deployment and MS Windows product license activation

In the past I have been informed from some of my customers that MS Windows Server license was not properly applied and activated  during VMware VM template deployment even the Product Key was properly entered in "Customization Specification".

I don't know if this issue still exists in the latest vSphere version however there was always pretty easy work around my customer is using since then.

You can use Run Once commands in "Customization Template". Below, in vSphere Web Client screen shot, you can see just example where two MS-DOS commands (dir and echo) are used.


For application and activation of MS Windows license key is leveraged tool slmgr.vbs which stands for "Windows Software Licensing Management Tool". 

Exact Run Once commands are:
  • C:\Windows\System32\slmgr.vbs -ipk H7Y93-12345-54321-ABCDE (just example use your product key)
  • C:\Windows\System32\slmgr.vbs -ipk

First command is used for product key application and second for activation.
Posted by at No comments:
Labels: , , ,

Wednesday, September 02, 2015

Storage related problems with LSI 12Gb SAS HBA card

Our Dell field engineer experienced strange storage problems with SAS storage connected to ESXi hosts having LSI 12Gb SAS HBAs. Datastores were inaccessible after ESXi reboot, paths were temporarily unavailable, etc. In this particular case it was DELL Compellent storage with SAS front-end ports but the problem was not related to particular storage and similar issue can be experienced on other SAS storage systems like DELL PowerVault MD, etc.

He has found (thanks Google) Christopher Glemot's blog post describing other issues with LSI 12Gb SAS HBA. However the key point was that mpt3sas driver should be used and not msgpt3 driver which was on these particular ESXi hosts.

The solution is relatively easy:

  • Uninstall msgpt3 driver - esxcli software vib remove –n lsi-msgpt3
  • Download the latest mpt3sas driver (currently mpt3sas-10.00.00.00-6.0-2803883.zip)
  • Unpack and upload mpt3sas driver  to ESXi host
  • Install  mpt3sas driver on ESXi host - esxcli software vib install –d /path/mpt3sas-10.00.00.00-6.0-offline_bundle-2803883.zip
  • Restart ESXi host
Hope this helps.
Posted by at 5 comments:
Labels: , , , ,

Wednesday, August 05, 2015

DELL Force10 : BPDU Guard vs BPDU Filtering

Spanning tree should be enabled on any enterprise switch during initial switch configuration. That's the reason I have mentioned spanning tree configuration in blog post "Initial switch configuration". On the latest FTOS version following spanning tree protocols are supported:

  • STP (Spanning Tree Protocol)
  • RSTP (Rapid Spanning Tree Protocol)
  • MSTP (Multiple Spanning Tree Protocol)
  • PVSTP+ (Per-VLAN Spanning Tree Plus)

I assume the reader is familiar with various spanning tree protocols and general difference between BPDU Guard and BPDU Filter. Here is just quick recap of relevant terminology
  • BPDU Guard and BPDU Filtering are Spanning Tree Protocol security mechanisms.
  • BPDU Guard is typically configured on particular switch edge port and it generally detects BPDU frames and because BPDU frames are not expected on edge port it disables the port temporary or permanently.
  • BPDU Filter is also typically configured on switch edge port and detects BPDU frames however it does not disable switch port but instead filter these BPDU frames to mitigate impact on spanning tree protocol because BPDU frames can initiate topology change and selection of STP root. 
Generally it is not recommended to filter BPDUs but there are at least some use cases where BPDU filter is beneficial. Here are at least two use cases in my mind where I believe BPDU Filter can be beneficial:
Use Case 1/ Datacenter interconnect (aka DCI) where you are absolutely sure there cannot be the loop and you want to have two independent spanning tree regions and you really want filter BPDUs
Use Case 2/ Edge ports to ESXi hosts
  • Rogue VM can send BPDUs to the network therefore some protection is needed especially in non-trusted  environments like IaaS Cloud Providers so you have to choose between BPDU Guard and BPDU Filter + Broadcast Storm Control
  • BPDU Guard can cause DoS when you don’t have control on ESXi configuration. See. http://blog.igics.com/2015/01/bpdu-filter-and-forged-transmit-on.html for further details 
  • BPDU Filter can help to mitigate topology changes when rogue VM is trying to be and not to be STP root switch periodically and initiate unwanted network topology changes   

This blog post main objective is explanation of specific DELL S-Series (formerly Force10) switches BPDU Guard and Filtering implementation so here it is.

Force10 BPDU Guard
  • Software-based implementation - BPDUs are received on an interface and passed to the CPU for analysis/action (logs will reflect dropped BPDUs)
  • Occurs when interfaces are configured for portfast/edge-port with bpduguard and a BPDU is received
Force10 BPDU Filtering
  • Hardware-based implementation - BPDUs are dropped on ingress to the interface
  • CPU does not receive BPDU, leaving CPU resources available for other tasks
  • Logs will not reflect dropped BPDUs because it is possible only when BPDUs are sent to the CPU
  • Occurs when STP is disabled globally or per-interface
Spanning Tree examples:
All examples below are for Rapid Spanning Tree Protocol (RSTP) but the same syntax works for other spanning tree protocol variants.

STP Portfast equivalent 
confinterface gigabitethernet 0/1  spanning-tree rstp edge-port
BPDU Guard - set port state to Error Disabled and drop traffic if a BPDU is received on interface - interface state remain up 
confinterface gigabitethernet 0/1  spanning-tree rstp edge-port bpduguard
BPDU Guard - shutdown interface if a BPDU is received on interface
confinterface gigabitethernet 0/1  spanning-tree rstp edge-port bpduguard shutdown-on-violation
BPDU Filter on particular interface
conf
interface gigabitethernet 0/1
  no spanning-tree

Posted by at 2 comments:
Labels: , ,

Tuesday, July 28, 2015

How you understand documenting Conceptual, Logic, Physical?

I have just read following question in Google+ "VCDX Study Group 2015"
As a fellow writer (we architects are not readers, but writers! :) ) wanted to ask you how you understand documenting Conceptual, Logic, Physical.
Can you add all these in a single Architecture design document with all 3 parts as 3 sections or you are better off creating 3 separate documents for each type of design?
I'm hearing very often similar questions about approach how to write good design documentation. So my answer was following ...

As a writer you have to decide what is the best for your readers :-)

When I'm engaged to write Architecture document I use different approaches for different design engagements. It really depends on project size, scope, audience, architecture team, etc... For example, right now I'm working on project where 6 architects are working on single High Level Design covering big picture and each preparing Low Level Design. At the end there is single HLD document and five separate LLD documents covering
  • Compute, 
  • Storage, 
  • Networking,
  • vSphere and
  • Backup.
I had another projects where whole architecture was in single document where each section was targeted to different auditorium. That was a case of my VCDX design documentation.

Generally I believe High Level Design (HLD) is for broader technical audience but also for business owners. Therefore physical design is not required in this stage and only Conceptual and brief Logical design for each area should be in HLD. Low Level Design (LLD) is for technical implementers and technical operational personnel therefore less writing creativity and more deep technical language for specific area should be used there with references to HLD. I recommend to read Greg Ferro's "Eleven Rules of Design Documentation" which IMHO apply very good to LLD.

HLD Conceptual Design should include business and technical requirements, constraints, assumptions, key design decisions, overall high level concept and risk analysis).

HLD Logical Design should include basic logical constructions for different design areas together with capacity planning.

LLD should include Conceptual, Logical and Physical design for specific area(s) or designed system/subsystem. In LLD conceptual design there should be a subset of HLD technical requirements, constraints and assumptions and maybe some other specific requirements irrelevant in HLD. They can be even discovered after HLD and LLD design reviews and additional technical workshops. Logical design can be the same as in HLD or you can go into deeper level but still stay in logical layer without product physical specifications, cabling port IDs, VLAN IDs, IP addressing, etc... These physical details should be in in physical design and if needed referenced in to attachments, excel workbooks, or similar implementation/configuration management documents.

LLD Physical design is usually leveraged by implementer to prepare As-Built documentation.

That's just my $0.02 and your mileage can vary. 

At the end I have to repeat ... you, as a writer (Architect), have to decide appropriate documentation format for your target audience.

Don't hesitate to share your thoughts in comments.
Posted by at No comments:
Labels: , ,

Tuesday, July 07, 2015

DELL Force10 : Interface Configuration and VLANs

Physical interface configuration

Physical switch interface configuration is a basic operation with any switch device. DELL Force10 switch is no exception. However, one thing is very unique on Force10 switches. Everything, including physical interfaces, on Force10 switch, is disabled by default, therefore, interfaces are in downstate and must be configured before any use. Someones are saying it is strange behavior but in my opinion, that's pretty good behavior because it is a much more secure approach. You will not disrupt the whole network by connecting and cabling new switch into your enterprise network until you configure something. If you will do bad configuration than it is your fault and not device fault.

Ok, so when you want to use some switch interface you have to enable interface explicitly. Before that, you should be absolutely sure your new Force10 switch is ready to be connected to the network. Think for example about spanning tree protocol configuration. Let's assume you know what you do and you want to enable the particular physical interface. It is easy. I think the example below is self-explanatory

conf
  interface tengigabit 0/1
  no shutdown

So your interface is up but another important note is that all physical interfaces are Layer 3 by default. You can assign IP address to Layer 3 (routed) interface and your L3 switch is configured as a router device. IP address assignment is shown below.

conf
  interface tengigabit 0/1
  ip address 192.168.1.11/24
  no shutdown

Cool, but there is a chance you want configure Layer 2 interface to work as a switch port and not routed port. It is pretty easy, you have to tell it to your interface to not have IP address and behave as a switch port.

conf
  interface tengigabit 0/1
  no ip address
  switchport
  no shutdown

Physical Interface Numbering

So far we have used similar interface identification as follows

interface tengigabit 0/13

General interface identification convention parts have following format

interface “Interface Type” “Stack Unit Number”/”Interface Number”
where

·       Interface Type - can have values gigabit (gi), tengigabit (te), fortygigabit (fo)

·       Stack Unit Number - is stack ID number if classic stacking is configured otherwise there is 0 as it is a single unit switch

·       Interface Number - is a sequential port number on particular stack unit


 Interface ranges

You can leverage interface ranges to simplify interface and VLAN configurations.

conf
interface range  te 0/1-3, te 0/5-7
interface range vlan 100-110
interface range vlan 4, vlan 11, vlan 22-26, vlan 100

I think you can see the benefit. All configurations are applied to all interfaces on the range.

VLANs

In configurations above everything is happening in default VLAN which is by default VLAN 1. This is a single broadcast domain. In computer networking, a single layer-2 network may be partitioned to create multiple distinct broadcast domains, which are mutually isolated so that packets can only pass between them via one or more routers; such a domain is referred to as a virtual local area network, virtual LAN or VLAN. Source: wikipedia. Nowadays VLANs are used very often for network separation (security) and broadcast domain split (availability, performance). If you are familiar with CISCO VLAN configuration then you have to create VLAN id in VLAN database and you can assign VLAN/VLANs to the particular interface. In Force10 it is a little bit different. You have to create VLAN id, that's the same. However, you are not assigning VLAN per interface but assigning interfaces to VLAN. See example below.

conf
  interface vlan 100
  description "VLAN for mission critical servers"
  untagged TenGigabitEthernet 0/1-3
  tagged TenGigabitEthernet 0/0

In the example above we have created VLAN 100 for three mission-critical servers. Servers are connected to ports Te 0/1, Te 0/2 and Te 0/3 without VLAN tagging (aka access mode in Cisco terminology). The particular switch port in "access" mode is configured in Force10 in following way

interface TenGigabitEthernet 0/1
  description "Mission critical server 1" 
  no ip address
  switchport
  spanning-tree rstp edge-port bpduguard 
  no shutdown

Switch interface Te 0/0 is uplink to the rest of the network, therefore, more VLANs has to be configured on this particular port, therefore, the port is configured as a trunk port with more tagged VLANs. Switch uplinks are usually configured redundantly in a high availability mode, therefore, there is a big chance you would like to use port-channel (aka LAG) as a switch uplink. LAGs are explained in next section. Port-channel is nothing else as a special virtual interface, therefore, port-channel VLAN configuration is very similar to physical interfaces.

conf
interface vlan 100
  description "VLAN for mission critical servers"
  untagged TenGigabitEthernet 0/1-3
  tagged Port-channel 1

There is another switch port mode which is typical for ESXi hosts. It is the server, but you want one VLAN to be configured as a native (usually for ESXi management) and also to configure trunk of multiple VLANs for virtual networking (VMware portgroups). Force10 call this port configuration as a hybrid. Switch port configuration will look similar as an example below

interface GigabitEthernet 0/11
  description ESX11
  no ip address
  mtu 9252
  portmode hybrid
  switchport
  spanning-tree rstp edge-port
  no shutdown

and for such hybrid switch port, we can have one VLAN configured as an untagged (aka native in Cisco terminology) and multiple VLANs as tagged. VLAN configuration should look similar to ...

interface Vlan 4
  description DC-MGMT
  ip address 192.168.4.254/24
  untagged GigabitEthernet 0/4,6,11-14,34-36,41,43
  no shutdown

VLAN 4 is a used for ESXi management but multiple other VLANs can be carried to the ESXi host as tagged VLANs. This is depicted in configuration snippet below ...

interface Vlan 22
 description VMOTION
 name VMOTION
 no ip address
 tagged GigabitEthernet 0/11-14,34-36
 shutdown
!
interface Vlan 23
 description VTEP
 name NSX-OVERLAY
 ip address 192.168.23.254/24
 tagged GigabitEthernet 0/11-14,34-36
 no shutdown
!
interface Vlan 24
 description ISCSI
 name ISCSI
 ip address 192.168.24.254/24
 tagged GigabitEthernet 0/11-14,34-36
 untagged GigabitEthernet 0/10
 no shutdown
!
interface Vlan 25
 description NFS
 name NFS
 ip address 192.168.25.254/24
 tagged GigabitEthernet 0/11-14,34-36
 no shutdown
!
interface Vlan 26
 description VSAN
 name VSAN
 no ip address
 tagged GigabitEthernet 0/11-14,34-36
 shutdown
!
interface Vlan 100
 description V2P-PEERING
 name V2P-PEERING
 ip address 172.16.0.254/24
 tagged GigabitEthernet 0/11-14,34
 no shutdown

So the solution above is one way how to do it. VLAN by VLAN, but what if I would like to configure two new ports into existing VLANs?

Let's assume I have two switch ports (gi 0/29 and gi 0/34) which I want to configure for ESXi hypervisor. Below is the basic configuration of switch ports.

interface GigabitEthernet 0/29
 description ESX01-nic1
 no ip address
 mtu 9216
 portmode hybrid
 switchport
 spanning-tree rstp edge-port
 no shutdown

interface GigabitEthernet 0/34
 description ESX01-nic0
 no ip address
 mtu 9216
 portmode hybrid
 switchport
 spanning-tree rstp edge-port
 no shutdown

Those, who are familiar with CISCO switch operating systems, they would expect VLANs configuration along with switch port configuration. Dell FTOS is different because you have to configure VLANs from the VLANs point of view and not from the switch ports point of view. Let's assume we have VLAN 4 for vSphere management network segment where ESXi hosts are connected natively without 802.1Q tagging. The rest of VLANs we would like to expose into ESXi must be tagged. These VLANs are 2-3, 5-9, 11, 13, 22-26, 31-34, 51-52, 100-101. So, below is the FTOS CLI commands to add two particular switch ports to several required VLANs ...

conf
interface vlan 4
  untagged GigabitEthernet 0/29
  untagged GigabitEthernet 0/34

interface range vlan 2-3,vlan 5-9,vlan 11,vlan 13,vlan 22-26,vlan 31-34,vlan 51-52
  tagged GigabitEthernet 0/29
  tagged GigabitEthernet 0/34

interface range vlan 100-101
  tagged GigabitEthernet 0/29
  tagged GigabitEthernet 0/34

Note: In this particular case, I have to use two ranges because the FTOS interface range is limited. See what error message you would get if you try to configure single interface range of all VLANs mentioned above.
 
interface range vlan 2-3,vlan 5-9,vlan 11,vlan 13,vlan 22-26,vlan 31-34,vlan 51-52,vlan 100-101
% Error: Exceeds maximum number of command arguments ( max = 32 ).

LAGs - Link Aggregates 

Link Aggregation is a general term for channeling multiple links into single virtual aggregate also known as a port channel. There are two types of port channels static and dynamic (aka LACP). For more general information about "link aggregation" look here.

Now let's see how you can configure port channels.

Static Port Channel
Below is the example of static port channel bundled with two interfaces (te 0/1 and te 0/2)

interface port-channel 1
  description "Static Port-Channel"
  channel-member tengigabit 0/1
  channel-member tengigabit 0/2
  no ip address
  switchport
  no shutdown

Dynamic Port Channel
Below is the example of dynamic port channel bundled with two interfaces (te 0/1 and te 0/2)

VLT (Virtual Link Trunking) is actually virtual Port Channel spanned across multiple chassis (aka MultiChassis LAG). VLT can be static or dynamic port-channel. When two Force10 switches are configured in single VLT domain you can create VLT port-channel independently on each VLT node. You can read more about VLT here.

You configure VLT port-channel on each node in the absolutely same way as classic port-channels. The only difference is that you will tell FTOS that this particular port-channel is VLT and you can define peer port-channel id which can be different than on another node. However, the best practice is to use same port-channel IDs on both VLT nodes just to keep configuration simple and more readable.

Directive to tell the port-channel is VLT is vlt-peer-lag.

So if port-channel examples above would be VLTs then the configuration is the same only with one additional option. See examples below.

Static VLT Port Channel

interface port-channel 1
  description "Static Port-Channel"
  channel-member tengigabit 0/1
  channel-member tengigabit 0/2
  vlt-peer-lag port-channel 1
  no ip address
  no shutdown

Dynamic VLT Port Channel

interface port-channel 1
  description "Dynamic Port-Channel (LACP)"
  no ip address
  vlt-peer-lag port-channel 1
  switchport
  no shutdown 

interface tengigabit 0/1
  port-channel-protocol lacp
    port-channel 1 mode active
  no shutdown 

interface tengigabit 0/2
  port-channel-protocol lacp
    port-channel 1 mode active

  no shutdown

Conclusion

Interface and VLAN configuration is a basic network operation. If you are familiar with any other switch vendor interface configuration I think Force10 interface configuration is simple for you. The only different approach is with VLAN configuration but it is just a matter of habit.

Hope you found this blog post useful and as always, any comment and feedback are highly appreciated.
Posted by at 4 comments:
Labels:
Subscribe to: Posts (Atom)