How to downgrade IBM V7000 (Storwize) firmware

Sometimes, especially when you do a problem management, you have a need to downgrade firmwares on some system components. I have such need for IBM V7000 storage array. Downgrade process is not documented in IBM official documentation so here is the downgrade process step by step:

1. Double check you have IP addresses on management interfaces of both canisters (controllers)
2. Login to management interface of one particular canister over https. https://[ip_mgmt_canister]. You have to use superuser credentials. Default IBM Storwize superuser password is passw0rd
3. Switch node to serrvice state. You should wait 15-20 minutes
4. Login to second node management interface. 
5. Switch second node to service state. You should wait another 15-20 minutes
6. Double check both nodes are in service state
7. Login to one node and choose action "Reinstall software". Browse and upload firmware image via web browser. Software reinstallation takes a while. You have to wait approximattely one or two hours. In the mean time you can ping canyster management IP addresses to check when nodes comming back. 
8. Repeat software reinstallation for second node.
9. Please be aware that storage configuration is lost after software reinstallation. Therefore you have to use default password for superuser. Recall it is passw0rd
10. When both nodes are up and running login to one canister node management interface and exit both nodes from service state. It can takes another 15-20 minutes.
11. When nodes are active you have to regenerate Cluster ID. You have to go to "Configure Enclosure" and enable checkbox "Reset System ID".
12. After all these actions you have Storwize ready to form a new Cluster. So create cluster and assign cluster virtual IP address you will use for standard storage management.

ESX host remote syslog configuration

For remote CLI you can use vMA or vCLI. Here is the example how to configure ESX host (10.10.1.71) to send logs remotely to syslog server listening on IP address 10.10.4.72 on tcp port 514. 

First of all we have to instruct ESX where is the syslog server.

esxcli -s 10.10.1.71 -u root -p Passw0rd. system syslog config set --loghost='tcp://10.10.4.72:514'

Then syslog service on ESX host must be restarted to accept configuration change.

esxcli -s 10.10.1.71 -u root -p Passw0rd. system syslog reload

ESX firewall must be reconfigured to allow syslog traffic

esxcli -s 10.10.1.71 -u root -p Passw0rd. network firewall ruleset set --ruleset-id=syslog --enabled=true
esxcli -s 10.10.1.71 -u root -p Passw0rd. network firewall refresh

If you want to test or troubleshoot syslog logging you can login to ESX host and use logger command to send test message to syslog.

logger "Test syslog over network"

Excellent article: "Anatomy of an Ethernet Frame"

Trey Layton (aka EthernetStorageGuy) wrote excellent article about MTU sizes and Jumbo Frame settings. The article is here. In the article you will learn what MTU size parameters you have to configure in the path among server, network gear and storage. It is crucial to understand difference between payload (usually 1500 or 9000) and different frame sizes (usually 1522 or 9018 or 9022 or  9216) on networking equipment.

Here is summation of  Trey's deep Ethernet frame anatomy in to the simple best practice. "If you want to implement Jumbo Frames use pure datacenter networking equipment and setup MTU size to the device maximum which is usually 9216."

DELL Open Manage Essentials 1.2 has been released

Dell OpenManage Essentials is a 'one to many' console used to monitor Dell Enterprise hardware. It can discover, inventory, and monitor the health of Dell Servers, Storage, and network devices. Essentials can also update the drivers and BIOS of your Dell PowerEdge Servers and allow you to run remote tasks. OME can increase system uptime, automate repetitive tasks, and prevent interruption in critical business operations.

It can be downloaded here.

Fixes & Enhancements
Fixes:

1. Multiple defect fixes and performance improvements

Enhancements:

1. Support for Discovery, Inventory and Map View for Dell PowerEdge VRTX devices. 
2. Addition of Microsoft Windows Server 2012 as a supported operating system for the management station.
3. Context sensitive Search functionality. 
4. Ability to configure OpenManage Essentials to send the warranty status of your devices through email at periodic intervals. 
5. Ability to configure OpenManage Essentials to generate a warranty scoreboard based on your preference and display a notification icon in the heading banner when the warranty scoreboard is available.
6. Enhanced support for Dell Compellent, Dell Force10 E-Series and C-Series, Dell PowerConnect 8100 series, Dell PowerVault FS7500, and PowerVault NX3500 devices. 
7. Support for installing OpenManage Essentials on the domain controller.
8. Device Group Permissions portal. 
9. Additional reports: Asset Acquisition Information, Asset Maintenance Information, Asset Support Information, and Licensing Information. 
10. Addition of a device group for Citrix XenServers and Dell PowerEdge C servers in the device tree. 
11. Availability of storage and controller information in the device inventory for the following client systems: Dell OptiPlex, Dell Latitude, and Dell Precision.
12. CLI support for discovery, inventory, status polling, and removal of devices from the device tree. 
13. Availability of sample command line remote tasks for uninstalling OpenManage Server Administrator and applying a server configuration on multiple managed nodes. 
14. Support for SUDO users in Linux for system updates and OMSA deploy tasks. 
15. Display of a notification icon in the heading banner to indicate the availability of a newer version of OpenManage Essentials. 
16. Support for enabling or disabling rebooting after system update for out-of band (iDRAC) system updates.
17. Support for re-running system update and OpenManage Server Administrator (OMSA) deployment tasks.
18. Support for Single Sign-On (SSO) for iDRAC and CMC devices. 
19. Ability to log on as a different user.

How to change default path selection policy for particular storage array?

Sometimes the firmware in storage array has some problems and you have to "downgrade" functionality to achieve operable system. That's sometimes happen for some ALUA storage systems where Round Robin path policy or Fixed path policy (aka FIXED) should work but doesn't because of firmware issue.

So relatively simple solution is to switch back from more advanced round robin policy to legacy - but properly functioning -  Most Recently Used path policy (aka MRU) normally used for active/passive storage systems. 

Note: Please be aware that some storage vendors saying they have active/active storage even they have not. Usually and probably more precisely they call it "dual-active storage" which is not same as active/active. Maybe I should write another post about this topic.

You can change Path Selection Policy by several ways and as always the best option depends on your specific requirements and constrains.

However, if you have only one  instance of some storage type connected to your ESX hosts you can simply change default path selection policy for this particular SATP type. Let's assume you have some LSI storage.

Below is simple esxcli command how to do it ...

esxcli storage nmp satp set --default-psp=VMW_PSP_MRU --satp=VMW_SATP_LSI

... and then your default PSP for VMW_SATP_LSI is now VMW_PSP_MRU

One thing you must be aware … if you in the past explicitly changed any devices (disks) to another path selection policy then default PSP will not change even you have another default path policy. There is not esxcli mechanism how to change devices back to accept  default PSP for particular SATP type. Only solution is to edit /etc/vmware/esx.conf

All previous explicit changes are written in /etc/vmware/esx.conf  so it is pretty simple to find it and remove these lines form config file.  I silently assume you do such operations in maintenance mode so after ESX reboot all your paths for your devices will follow default SATP path selection policy. 

BTW: That’s why I generally don’t recommend to change PSP for particular device when not necessary. Sometimes it is necessary for example for RDM’s participating in MSCS cluster. But usually it is abused by admins and implementation engineers.  I strongly believe it is always better to set default PSP to behave as required.

Do you want to test Heavy Load? Try Heavy Load tool.

Bring your PC to its limits with the freeware stress test tool HeavyLoad. HeavyLoad puts your workstation or server PC under a heavy load and lets you test whether they will still run reliably.

Look at http://www.jam-software.com/heavyload/