I had a customer recently buy 2 Cisco Nexus 5548UP switches to be used for their Fibre Channel (FC) Storage functionality, which is 50% of the total switch functionality. For Ethernet features, the customer bought separate Nexus switches. I did not advise on the design of this solution. However, it is a telling sign of Cisco’s weakened efforts to play down native FC functionality in favor of native Fibre Channel over Ethernet (FCoE). Unfortunately, promoting FCoE is proving to be harder than it looked. This is due to two main problems — confusion about the management of FCoE and Cisco’s main storage switch competitor Brocade wielding formidable muscle to keep native FC alive.

 

In 2003, Cisco entered the FC native switch market with a splash. Cisco did this by perfecting one of their first spin-ins. A spin-in is when a company tells a few hundred of their engineers, “You will become a separate company and receive shares. Then, you will build us the best new technology on the market. If the tech is successful, Cisco will buy the company back, and all who received shares will profit handsomely.” Thus, Andiamo Systems was born and built the Cisco MDS FC storage switch.

 

At the time of their MDS FC switch release, Cisco’s primary competitor Brocade had 1/3 of the software features of the new MDS switch. I remember talking to my colleagues, old school FC guys, about the Cisco MDS. “Cisco does not know the FC storage market — they are network specialists. They will never release a switch as easy and robust as a Brocade or McData,” forecasted my bearded comrades. I believed them — they had years of experience.

 

Soon, Cisco was sending me to MDS courses, and the new CCIE certification in Storage was born. After I took the courses and realized the MDS’s feature dominance, I was hooked. Any geek worth his salt likes an abundance of gadgets — Cisco had Virtual Storage Area Networks (VSANs), FC pings, Inter-VSAN routing and Internet Small Computer System Interface (iSCSI) built in. This was when iSCSI storage was a novel, branch only solution. But with Cisco, you could do iSCSI with ANY storage.

 

The only Cisco problem was cost. Cisco attacked the large enterprise market first. As a result, an entry level MDS 9216 switch with all the wizardry cost $52 thousand list. The price was prohibitive for smaller markets. However, Cisco’s access to large enterprise decision makers allowed the company to quickly dispatch of the McData switch company. McData was gobbled up by Brocade. Brocade was not passive — it quickly ramped up development of software features. Further, Brocade’s main counter bet was the raw hardware speed strategy. Brocade decided to improve native FC chips faster than Cisco. As a result, Brocade was faster to market with 4 Gbit and then 8 Gbit FC.

 

At the time of Brocade’s 4 and 8 Gbit increases, most critics said that no client requires such speed, that it is purely a marketing gimmick. Such marketing gimmick charges were also leveled against Cisco for advanced software gadgets in their switches. The FC customer had a choice — raw FC speed vs. advanced software. While I enjoyed advanced software, I admit — few clients used switch based iSCSI and extra Cisco features. Both Brocade’s FC speed dominance and Cisco software advantage were pure marketing. But with time, it became evident that faster chips prevail in the marketing argument. Software was faster to develop than custom Application-specific Integrated Circuits (ASICs). Consequently, Brocade matched Cisco’s features, and was winning in FC speeds. Pundits that shot down the speed were silenced, as throngs of customers wanted faster Input/Output (I/O) for their growing VMware virtual server farms.

 

Still, Cisco had another card up its sleeve. Andiamo’s success was spectacular. The team behind the MDS was itching to disrupt another market, and Cisco was happy to oblige. Nuova Systems put together the same cast of characters in another spin-in. The spin-in secretly burrowed inside Cisco, building something unique. Speculation abounded, but when Cisco finally announced Nuova’s work, it was revolutionary.

 

Cisco/Nuova was the first to release the Nexus, a unified switch which supported native FC and Ethernet/Internet Protocol (IP) in the same box. Further, Cisco was the first to release the 10 Gbit FCoE protocol for their unified switches. Also, Cisco entered the server market with the Unified Computing System (UCS), a blade server platform with a network oriented architecture. Brocade was again caught off-guard, and went to its tried and true strategy — win with native FC, while countering Cisco’s feature superiority with “me toos.” Thus, Brocade was the first on the market with 16 Gbit FC. Meanwhile, Cisco touted the benefits of unified switching at 10 Gbit.

 

The promise of unified switching made complete sense. Why have 2 separate networks — storage and IP, which have similar concepts? Yet, in the 1990s, the storage market diverged onto a separate network path. Storage required a switching infrastructure, and the industry delivered the Storage Area Network (SAN). Developed in part by the SCSI guys mixed in with IP pros, FC was a robust protocol, perfect for storage. However, when FC expanded into multiple sites and routing, its management became similar to IP network management. Still, storage was managed by the storage team and IP by the network team. Separate, sometimes warring Network vs. Storage silos increased hardware and staff costs at large enterprises.

 

Cisco argued it could reintegrate the silos into one job role — an omniscient super Data Center guru. To that end, Cisco Certified Internetwork Expert (CCIE) certification in Storage was recently discontinued in favor of CCIE in Data Center. The Data Center CCIE requires knowledge of the server in Cisco UCS, the storage of Cisco MDS and Nexus FC, the Cisco Nexus Layer 2/3 network, and the Nexus 1000V virtual switching functionality. In Cisco’s view, the Data Center CCIE is the James Bond of the Data Center world, able to handle any problem thrown at him. I have no doubt that such people will exist, and they will assemble multi-disciplinary departments.

 

However, the current realities in the field seem to tell a different story. Over time, FC and storage developed into a whole separate area with its own deep expertise. The storage professional was required to know the management of multiple storage arrays from different manufacturers — EMC, NetApp, HP, Hitachi, Dell, and others. The number of storage protocols increased from just FC and occasional Fibre Channel over IP (FCIP) for routing across distances to iSCSI, Network File System (NFS), Common Internet File System (CIFS) and FCoE. Brocade absorbed McData, but Qlogic began to make FC switches.

 

On the network side, the amount of technology keeps increasing. The Routing and Switching CCIE of today has to know 3-5 times more than CCIE #1. Moreover, we have virtual and Software Defined Networking (SDN) disrupting the field. The future network guy has to know FabricPath, Overlay Transport Virtualization (OTV), the new Cisco’s Cloud Services Router 1000V (CSR), VMware’s vCloud Director, Nicira, and Brocade-acquired Vyatta.

 

The FCoE protocol was meant to unify storage and network management through the Nexus switch. Unfortunately, what I see is that the network guys are weak in the Nexus FC functionality. Yes, they may have gone to a class to learn the MDS or Nexus’s FC side, and they know the transport side. However, they lack the knowledge and control of the day to day management of the endpoints — server, virtualization and storage arrays. As a result, whoever controls the Nexus FCoE, cannot control and troubleshoot the native NetApp FCoE card, and also cannot control LUN presentation in a UCS blade. But, complete control of the FC stack is essential in FC troubleshooting. On the other hand, storage guys rarely want to delve into the network features of the Nexus switch. Consequently, because the Nexus is first and foremost a network switch, the storage guys never have daily administrative control.

 

When a shop is moving from one manufacturer to another, toward a converged network, it does not want to have multiple manufacturers. Another customer asked me to design a Nexus only solution, because they wanted to move away from Brocades. They had money for the Nexuses, but not enough Nexuses to accommodate all Brocade FC ports currently in existence. When I mentioned introducing Cisco MDSs, they said, “Why are you introducing native FC into the mix when the Nexus FCoE mantra is suppose to solve our storage needs?” The death of native FC, and especially Cisco MDSs has been predicted before. When Cisco retired CCIE in Storage, that was another sign that the MDS native FC switch is on its way out. Yet, Brocade released 16 Gbit native FC 2 years ago.

 

Today, the Data Center has many management headaches due to convergence.

• Who will manage the server and its network devices (Converged Network Adapter cards, end-host I/O modules like Hewlett-Packard Virtual Connect or Cisco UCS Fabric Interconnects) — VMware guys, old school server guys, storage or network gals?
• Who will manage the network going from the server out to the first access device (end-host I/O modules, Cisco UCS Fabric Interconnects)?
• Who will manage storage and network transports (Cisco Nexus, Brocade switches)?
• Who will manage the network in a virtual network world (Cisco Nexus 1000V, CSR, Vyatta, Nicira, hardware Nexus, OpenStack)?

These management questions have not been settled. In fact, what I see is the expertise of IT staff, like water, flows back and forth between departments. No one is quite sure where her responsibility ends and the colleague’s begins.

 

In this world of management confusion, the tried and true resonates. As a result, Brocade was successful in convincing the Data Center to wait on the death of native FC. And, just like before with 4 and 8 Gbit FC, Cisco had to answer. Therefore, the new Cisco 16 Gbit FC MDS 9710 Multilayer Director and the upcoming MDS 9250i Multiservice Fabric Switch that were just announced continue the time honored tradition of Brocade vs. Cisco FC war. In response, on LinkedIn Brocade immediately pointed to a press release touting added software features in their current code base. In addition, I am sure Brocade R&D is already well on the way to release the 32 Gbit FC ASICs in the next round of battle.

 

Meanwhile, many IT departments like my original customer will continue to follow a dual path — native FC separate from the Network, avoiding FCoE. Even when that avoidance is done on the switches from the same manufacturer — Cisco, and when it happens with the switch that fully supports FCoE — the Nexus. The native Fibre Channel protocol is alive and well.

 

References:

Cisco Helps Customers Address Rising Cloud, Big Data Requirements By Raising the Bar for Storage Networking.

Brocade Advances Fibre Channel Innovation With New Fabric Vision Technology.