Storage design verification - performance test

I had a unique chance to work with relatively big customer on VMware vSphere Architecture Design from the scratch. I prepared vSphere Architecture Design based on their real business and technical requirements and the customer used the outcome to prepare hardware RFI and RFP to buy the best hardware technology on the market from technical and also cost point of view. Before design I did capacity and performance monitoring of customer's current environment and we used these numbers for capacity sizing of new infrastructure. I designed the logical hardware architecture of fully integrated compute/storage/network infrastructure blocks (aka PODs - Performance Optimized Datacenter) where PODs are leveraged as vSphere Clusters with predefined and well known performance characteristics and ratios among CPU, memory, storage and network.

We all know the most complicated is storage performance sizing especially with leveraging automated storage tiering technology existing in almost all modern storage systems. I was able to prepare some estimations based on standard storage calculations and my experience however we left final responsibility on hardware vendors and their technical pre-sales teams. Our requirement was pretty easy - 60TB of capacity and 25,000 IOPSes generated from servers in R/W ratio 70/30.

Validation and acceptance test of storage was clearly defined. The storage systems must be able to handle a 25,000 IOPS workload synthetically generated leveraging free tool IOmeter. Test environment was composed from 250 linux VMs with single Worker (IOmeter dynamo). All these workers were connected to single IOmeter GUI reporting total workload nicely in single place. Each of 250 workloads were defined as described below:

• Outstanding IO: 1
• IO size: 64KB
• Workload pattern Random/Sequence ratio: 70:30
• Read/Write Ratio: 70:30

Hardware vendor was informed that we will run this workload during 24 hours and we want to see average performance of 25,000 IOPSes with response times below 25 ms.

Selected hardware vendor delivered storage with following disk tiers:

• Tier 1: 8x 400GB 6G SAS 2.5” MLC SSD R5 (3+1)
• Tier 2: 128x 300GB 6G SAS 15K 2.5” HDD R5 (7+1)
• Tier 3: 40x 900GB 6G SAS 10K 2.5” HDD R5 (7+1)

We asked hardware vendor how LUNs for vSphere datastores should be created to fulfil our capacity and performance requirement. The vendor recommended to leverage automated storage tiering and stretch the LUN across all disk tiers. We were able to choose particular disk tier for first write into LUN. It was selected to Tier 2. It is important to mention that process for automated storage tiering runs by default one a day and it can be changed. However from my experience it is usually even worse because if you generate continuous storage workload and AST background process starts then it generate another workload on already highly loaded storage and response times becomes unpredictable and sometimes it makes even bigger problem. AST is good technology for typical enterprise workloads when you have good capacity and performance ratio among Tiers and you have tiering window when your storage is lightly loaded and you can run AST background process to optimize your storage system. It's important to mention that AST requires really good planning and it is not good technology for continuous stress workload. But that's what hardware vendor pre-sales team has to know, right?

The result where we are right now is that we are able to achieve 15,600 front-end IOPSes which can be simply recalculated into back-end IOPSes based on read/write ratio and write penalty which is 4 for RAID 5. On figure below is screenshot from IOmeter just for illustration but final achievement was really 15,600 IOPS average from the beginning of the test.

Backend IOPSes = 10920 reads + ( 4 x 4680 writes) = 29,640 which can be recalculated into IOPSes per disk = 29640/128 = 231 IOPS. 231 IOPSes per 15k/rpm disk is pretty high and other Storage tiers are not leveraged so we are calling hardware vendor and asking how we can achieve our numbers.

BTW: this is acceptance hardware test and vendor has to prove this numbers otherwise he has to upgrade his storage (at his expense) or take the hardware out and return money.

To be continued ... stay tuned ...

UPDATE: Long story short ... at the end of the day storage vendor had to add additional disk enclosures with more spindles. Storage vendor had to pay it and it is worth to mention that it was significant additional cost covered 100% from his margin!!! No additional single cent paid by my customer. It is just another reason to engage subject matter expert for Infrastructure Design because when logical infrastructure design along with test plan is prepared before RFI and RFP your RFP strict requirements can be properly written and clearly articulated to all tender participants.