AuthorsB. Bogdanski, F. O. Sem-Jacobsen, S. Reinemo, T. Skeie, L. Holen and L. P. Huse
EditorsX. Huang
TitleAchieving Predictable High Performance in Imbalanced Fat Trees
Afilliation, Communication Systems
Publication TypeProceedings, refereed
Year of Publication2010
Conference NameProceedings of the 16th IEEE International Conference on Parallel and Distributed Systems
Date PublishedDecember
PublisherIEEE Computer Society
ISBN Number978-0-7695-4307-9

Abstract-The fat-tree topology has become a popular choice for InfiniBand fabrics due to its inherent deadlock freedom, fault-tolerance and full bisection bandwidth. InfiniBand is used by more than 40% of the systems on the latest Top 500 list, and many of these systems are based on a fat-tree topology. However, the current InfiniBand fat-tree routing algorithm suffers from flaws that reduce its scalability and flexibility. Counter-intuitively, the achievable throughput per node deteriorates both when the number of nodes in a tree decreases or when the node distribution among leaves is nonuniform. In this paper, we identify the weaknesses of the current enhanced fat-tree routing algorithm in OpenFabrics Enterprise Distribution and we propose extensions to it that alleviate all performance problems related to node distribution. The new algorithm is implemented in OpenSM for real world evaluation and for future contribution to the OpenFabrics community. We demonstrate that our solution allows to achieve a predictable high throughput regardless of the number of nodes and their distribution. Furthermore, the simulations show that our extensions improve throughput up to 30% depending on topology size and node distribution.

Citation KeySimula.netsys.16