Evaluation of 40 Gigabit Ethernet technology for data servers

<!--HTML-->40Gb/s network technology is increasingly available today in the data centers as well as in the network backbones. We have built and evaluated storage systems equipped with the last generation of 40GbE Network Interface Cards. The recently available motherboards with the PCIe v3 bus provide the possibility to reach the full 40Gb/s rate per network interface. A fast caching system was built using 16 SSD drives in a single server. The single-node system has been designed for disk data throughput at full 40Gb/s. We have evaluated data transfer performance in the data center environment using 40GbE switches. The last step in the evaluation was the demonstration, during SuperComputing 2011, of 40Gb/s disk-to-disk data throughput between a pair of servers over close to 4000 km WAN circuit. We review our experience with 40GbE technology in the LAN and WAN environment. We describe the system design, tuning performed, and the performance achieved. The system described has potential application as a caching or front-end system to a large, conventional, storage system, allowing fast data movement over high-capacity network channels. Such a system is of particular interest in combination with dynamic bandwidth reservation systems, as it allows efficient use of network resources available during the reservation period.