Jack, M_High-Performance Computing at Florida A&M University

In a rapidly changing technological landscape with ultrafast developments in science and engineering, a team of faculty across the Physical Sciences, Engineering and the School of Busine

High-Performance Computing at Florida A&M University

Author: Dr Mark A Jack, Associate Professor, Florida A&M University, Department of Physics,

FH-SRC 301C, Tallahassee, FL 32307 Phone/Fax: 850-599-8457/-3577 Email: mark.jack@famu.edu

High-Performance Computing Learning Community

High-performance with high-throughput computing (HPC) is becoming one of the key, truly transformative technologies of the 21st century Multi-core processing and parallel execution of software tools on modern computer architecture have become commonplace In a rapidly changing technological landscape with ultrafast developments in science and engineering, a team of faculty across the Physical Sciences, Engineering and the School of Business and Industry (SBI) at Florida A&M University (FAMU) are working together to advance HPC training and resources for students and faculty across disciplines on campus With FAMU as a designated Historical Black College and University (HBCU) of 13,200 students, 92% of which are African-American students and belong to historically underrepresented minorities in the STEM disciplines, such a program could pose a truly transformative force on campus and beyond to help create a highly skilled workforce to tackle complex systems analysis prevalent in most scientific and engineering challenges of today This team of faculty would like to formulate a strategy that teaches students an optimal set of core parallel computational skills coupled with entrepreneurial skills development in a joint, interdisciplinary learning environment It will create a versatile learning community of STEM and business majors who will be optimally prepared for projects using high-performance computing (HPC) in graduate school and industry Hands-on research projects in advanced computational modeling will be coupled with a discussion of real-life applications in industry and technology and train joint cohorts of undergraduate and graduate students in engineering, physics, chemistry, computer and information science, math and business and industry Students will learn to communicate effectively across disciplines and create innovative solutions in team settings on projects that integrate quantitative modeling in science, engineering and business with entrepreneurship training

Prof Tiki L Suarez-Brown’s Impact and Legacy in High-Performance Computing

A major driving force in promoting and advancing high-performance computing at Florida A&M University was the late Dr Tiki L Suarez-Brown, scholar and student mentor at FAMU’s School of Business and Industry and at departments across campus and who posthumously was awarded the promotion to full professorship by her peers for her accomplishments and lasting legacy Faculty and students were privileged to witness Professor Suarez-Brown’s – affectionately simply known as ‘Tiki’ by faculty and students alike – dedication towards advancing students’ learning and professional growth in

computing across campus and in organizations like the Supercomputing Conference, the Richard Tapia Conference in Computing and the Grace Hopper Conference for Women In Computing She demonstrated

a unique vision and zeal to select the technical arena of HPC as the main technology platform to train business and STEM majors alike, on- and off-campus, by building collaborative learning environments

between students and scholars Dr Suarez-Brown played a number of leadership roles on the ACM/IEEE

SC conference committee, including the SC Student Cluster Competition In all of these roles, she

engaged emerging communities in high performance computing: Students, people from traditionally under-represented groups, Minority Serving Institutions, and women Suarez-Brown served e.g as the SC12 and SC09 Broader Engagement (BE) Program Chair, SC11 and SC08 Student Volunteers Co-Chair, SC10 Student Cluster Competition Communities Chair, and SC03 SC Global MSI Outreach Coordinator

She had been a participant in the SC Minority Serving Institutions program for a number of years and played an important role in the evolution of that program into the Broader Engagement program At SBI,

Tiki Suarez-Brown was focused on engaging business students in technology where she taught all of her students the importance of business professionals who understand technology and that the two disciplines are intertwined In her ‘Systems Theory & Design’ course, students initiated and designed practical information systems projects These required them to utilize business and professional skills to solve technology systems problems or to introduce IT solutions to solve practical issues She also acted as an advisor and mentor to several business majors and authored multiple papers on collaborative

environments with her research students, some of which were presented at meetings such as the

International Conference on Industry, Engineering, Management and Systems

Research and Training in High-Performance Computing across Departments:

Dean Prof Shawnta Friday-Stroud and Colleagues – School of Business and Industry

The mission of Florida A&M University’s School of Business and Industry is to produce graduates capable of excelling as future leaders in global business, industry, and commerce In order to facilitate this growth SBI utilizes electronic tools to assist with its course management and delivery functions, which was initiated and advanced in SBI by Dr Tiki Suarez-Brown and colleagues The Professional Leadership Development Program in SBI for example benefits immensely by employing these modern technologies where the tools provide alternative methods to enhance students’ communication skills and

help them develop ubiquitous access to quintessential information The analysis of Access Grid

technology and similar advanced communication and computing technology with regards to boosting student performance in the classroom and in distance-learning environments while also advancing professional leadership development were some of the central research areas promoted by Dr Tiki Suarez-Brown and are now being continued under Dean Prof Friday-Stroud’s leadership See e.g.:

1 Suarez, T (2007) Access Grid Technology in Classroom and Research Environments Special Issue

on High Performance I/O Architectures and Systems J of Supercomputing, Vol 41(2), p 133-145

2 Suarez, T., Friday-Stroud, S and Hill, A (2006) Students’ Perceptions of Access Grid Technology as

a Synchronous Distance Learning Tool 3rd Annual Management Faculty of Color Association

Conference (MFCA), Tallahassee, FL

This research has led to a number of grants in SBI, most prominently of which the Summer Institute for Entrepreneurship, Science, Technology, Engineering and Math (E-STEM) and the Proctor & Gamble Curriculum Development Grant Program have to be mentioned, both sponsored by Proctor & Gamble

Inc with Suarez-Brown as a co-principal investigator in each case The first grant entails both a student leadership boot camp and an Entrepreneurship, Science, Technology, Engineering and Math (E-STEM) summer institute to expose students to and elevate student interest in entrepreneurship for STEM areas, while the latter led to the development of the new course ‘Dynamics of Global Information Management’

In particular the E-STEM initiative is now perfectly poised to integrate the efforts of jointly training SBI, computer and information science, physical science and engineering majors in the areas of advanced computing, modern information technology and entrepreneurial skills development and to create a unique synergy of learning in science, technology and business Algorithms and methods in the hard sciences and engineering could be employed to test complex transactions in certain business networks, or science and engineering majors could learn how to develop strategies to translate research results into small business models for commercialization In this context, one also has to point out Tiki Suarez-Brown’s outstanding

effort of bringing Access Grid technology to the campus in 2003/04 – a unique facility that provided training and education with advanced computing technology for students and faculty supported by Title III funding, NCSA and Florida Lambda Rail (FLR) This new facility allowed for first remote

visualization of large data sets to teach new STEM content to larger student cohorts with multi-institutional, interactive video- and web-conference interactions in a classroom setting via the big screen Housing a larger computer cluster at FAMU’s Media Center together with the Access Grid technology could ideally create a center and a shared resource on campus where students across disciplines could learn about advanced computing paradigms and technology in courses and projects with faculty

Dr Jesse Edwards – Department of Chemistry

Through collaborations between Dr Edwards at Florida A&M University’s Chemistry Department and Adrian Roitberg of the University of Florida’s Quantum Theory Project, students in chemistry will be able to participate in specifically designed computational projects under Edwards’ guidance, advisement and mentoring The Edwards group has published work with researchers at FAMU specializing in

anti-inflammatory steroidal compounds that target the estrogen receptor A in vitro where the Edwards team

has the longest molecular dynamic simulations of the estrogen receptor to date, with on-going studies

Students will receive training in two distinct areas of biomolecular simulations: Small molecule SERM’s

will be examined by use of molecular mechanics to examine rotational barriers of the flexible regions of the molecules that are believed to be the key points of activity in the compound’s anti-tumor behavior This will then be coupled with molecular dynamics simulations and continued docking studies on the ligand-binding domain of the estrogen receptor, of individual SERM’s and of co-peptide substrates

Edwards has also been collaborating with Ben Dunn (UF), regarding HIV-1 protease The Edwards-Roitberg team has produced the first ever simulation of the HIV-1 protease type C from a crystal structure

obtained from the Dunn laboratory (See Fig 1) In addition, Scott Shell of the University of California at Santa Barbara teamed with the Edwards and Roitberg groups to produce valuable conformational comparisons between HIV-1 protease B vs C Therefore, the application of data retrieved from the dynamic studies of the proteases will produce new pharmaceutical compounds to fight this currently drug- resistant form of the virus

In another project that combines experiment with computation, Edwards and a collaborative team from

the University of California at Santa Barbara (Craig Hawker et al) and the Materials Research Facilities Network (MRFN) have designed and synthesized micelles that tether cholesterol to poly-ethylene glycol

by way of a linker In an effort to examine how these bipolar molecules aggregate to form micelles a collaborative team of Shell, Edwards and Roitberg plan to do long-time molecular dynamics simulations

to examine properties of these polymeric systems With the use of large-scale computational resources whole micelles can be simulated using similar methodologies

Dr Hongmei Chi – Department of Computer and Information Science

Dr Hongmei Chi’s research interests in the Department of Computer and Information Science span many areas related to information security, parallel computing, and Monte Carlo simulations, including the design and analysis of parallel and distributed algorithms applied to bioinformatics, the design and analysis of parallel quasi-random sequences, and tools and libraries for general Monte Carlo simulations With the rapidly growing evidence that various systems in nature and society can be modeled as complex networks, community detection in networks has become increasingly important recently Community detection algorithms aim at detecting groups of nodes within which connections are dense in otherwise sparsely connected networks Most of the algorithms to achieve this task up to now have not been suitable for very large networks because of their complexity In Chi’s research, data visualization tools, such as

Vizster, are used to visualize a large amount of data from social media (See Fig 2) Implementing such

visualization and analyzing such a large dataset can only be completed on an HPC platform A new class

of patient-driven health social networks is emerging to supplement and in other cases extend traditional health care delivery methods and empower patient self-care Using data mining techniques, information

visualization, and Particle Swarm Optimization (PSO), the use of visualization tools will help reveal inner

patterns of health social networks The potential exists to both improve traditional health care systems and expand the concept of health care through new social media Additionally, students may compare different graphical methods to visualize in collaborations with Chemistry and Physics anything from

bio-Figure 1 Shows overlaid snapshots of simulations of the HIV-1 Protease C flap positions covering over 30 nanoseconds This is the 1 st simulation of the C type of HIV-1 Protease C known in the world

molecules to angular magnetic field distributions generated by currents in nanostructures Computational

tools such as Paraview and Visit would be optimally suited to visualize these systems on an HPC cluster

Drs Simon Foo, Petru Andrei, and Pedro Moss – Department of Electrical and Computer Engineering

With ECE Chair Dr Simon Foo and his colleagues, students will design experiments to measure the quantum of electrical conductance and apply these to the measurement of single atom electrical contacts Students will be introduced to peta-scale simulations of nanoelectronic devices The use of high- performance computing (HPC) will help address two challenges with the development of next-generation nanotransistors, namely (i) the capability of modeling realistically extended structures on an atomistic basis (Fig 3), and (ii) predictive simulations that are faster and cheaper than experiments

Multidimensional, quantum transport solvers from http://nanohub.org will be used to achieve these goals

P Andrei’s group will use the software RandFlux to simulate 3-D energy storage devices such as

lithium-air batteries, lithium-ion batteries, and fuel cells

This approach requires the solution of large systems of equations that will be performed on high-performance computing systems The students will have a chance to optimize the geometry and structure

of the energy storage devices by using cathodes with variable porosity and variable catalyst distribution Combined diffusion and mass transport equations that incorporate degradation mechanisms including

material phase change will be simulated in Matlab/Simulink Modeling should provide accurate results

and reveal new insight and a comprehensive understanding of how electrochemical storage devices work

a.

Figure 2: (a) Flow chart on data visualization (b) Visualization

of social network data

Figure 3 A monolithic tandem design of triple

junction solar cell comprised of

GaP/InGaAs/InGaSb

The simulation will make use of a simulation platform developed in-house at the FAMU-FSU College of Engineering, which will interact with the high-performance computer cluster to simulate the mass, ion, and electron transport in energy storage devices The operation of these devices is mainly governed by the transport of the lithium, oxygen, hydrogen, water, lithium hydroxide, and hexafluorophosphate in the porous cathode, in which the scale of the pores is

of the order of 10-20 nm The simulation of transport in the pores is performed by discretizing the main equations on a finite element grid, which should be sufficiently refined in order to capture the microscopic effects of the pores

Drs Mario Encinosa, Mark Jack, and Ray O’Neal – Department of Physics

Carbon nanostructures are the subject of much theoretical and experimental focus because they are thought to potentially be constituents of next-generation electronic devices A parallel object-oriented C++ code now exists in the working group of M Encinosa and M Jack where a generalized tight-binding code describes transport in nanotori attached to metallic leads to allow for the inclusion of static externally applied fields and arbitrary lead placement The project is originally based on a simpler, serial code developed by Encinosa at NASA Ames (2000) for limited system sizes of up to 4000 atoms M Jack

hosted two undergraduate summer students supported via the NCSI/Shodor Undergraduate Petascale Computing and Education Program (UPEP) and NSF TeraGrid in summers 2010 and 2011 to develop

the new parallel tool with much larger functionality and allowing for simulation of system sizes of several 10,000 atoms on NSF XSEDE resources All transport observables can be derived from the Green’s

function G d (E) of the device region in a non-equilibrium Green’s function approach As a possible student

project, the conductivity through nanotori as a function of lead placement or the inclusion of next-to-nearest neighbor interactions could be studied, made possible through the use of general parallel sparse

matrix libraries like PETSC Figure 4 illustrates the general kinematics and mappings used to develop a

continuum model of an example nanostructure – in this case a toroidal carbon nanotube In a second step, time-dependent methods would be employed to calculate the influence on transport arising from the interplay between phonon modes and curved regions inherent to the structure and regions of curvature induced by those modes

In Dr O’Neal’s group, students will have the opportunity to inspect, using an SEM, a device composed of

a thin film metal pattern deposited on a flexible substrate, which includes a constriction feature at nanoscale dimensions Placing a voltage across the device while measuring current, students will bend the device until the junction at the center of the device represents a single atom contact Students will analyze

the I-V curve obtained to determine where the quantum conduction behavior becomes evident and

estimate the quantum of conductance These measurements will be compared with numerical simulations

of charge propagation in a metallic film using Monte-Carlo simulations of the underlying multi-particle quantum theory, which are currently being run in Dr O’Neal’s lab on a small 20-processor core

mini-cluster and on Open Science Grid resources Students will investigate the Monte-Carlo modeling results

for the electronic characteristics of the metal film in the limit of a quantum point contact Band structure effects, influence of source-drain and gate potentials, materials defects, phonons and similar may be incorporated in the model by e.g adding effective potentials to charge transport across the point contact

Figure 4 (a) Continuum computation approach used to study reduced-dimension materials Intrinsic basis vectors and the use of representative area elements to sample inter-atomic potential energy allow the phonon spectra to be accurately predicted (b) Typical phonon mode predicted using the continuum approach

Faculty Case Study – HPC involvement of M Jack (Physics, FAMU)

History of HPC Access and Use:

Preliminaries:

2005/06 CNF Fall and Harvard U workshops on parallel tools (DFT, MD, FDTD etc.)

SC Conferences:

HPC intro, networking, community building, great student recruitment tool

• SC’06 Tampa, FL - Broader Engagement / BE

• SC’07 Reno, Nevada - BE

• SC’08 Austin, TX

• SC’10 New Orleans, LA - TeraGrid / SC BE (my group: 1 postdoc, 4 grads, 2+6 undergrads, from different institutions – FAMU, CAU, Morehouse, JSUMS, Winona State U.)

• SC’11 Seattle, WA - Education

Air Force / ASEE Summer Programs:

2009 and 2011 ASEE Summer Faculty Award, AFRL/RX, Dayton, OH

TeraGrid:

First parallel coding since 2010 (C++, ScaLAPACK; since 2011: MPI, PETSc)

• TeraGrid User Workshops TACC (2007), NCSA (2008)

• TeraGrid User Startup Accounts on TACC Ranger etc (2007-2010)

• 2009 TACC Supercomputing Institute

• 2010 TeraGrid Fellowship + Undergraduate Petascale Education Program (UPEP/Shodor)

Summer Students Leon W Durivage (2010), Adam Byrd (20110

SSERCA:

All code development and runs on SSERCA resources since fall 2010

• FL LambdaRail WS (Miami, July 2010), SSERCA HPC Summits 2011

• FSU HPC and U Miami ‘Pegasus’ accounts courtesy J Wilgenbusch and N Tsinoremas

+ J Zysman

Current HPC Activities:

• XSEDE research allocation (XRAC), Jul 2012 – Jun 2013:

Electron-phonon coupling in carbon nanorings, 10,000+ atoms production runs on TACC

‘Ranger/Stampede’

• Code development and benchmarking on SSERCA resources (FSU HPC, U Miami HPC, UCF IST) and on GPU cluster (Clark Atlanta U.)

• SC BE Committee and SSERCA HPC Summits since 2010

• XSEDE outreach: ITM, Tampa, 2012; XSEDE13, San Diego, Jul 2013; SC13, Denver, Nov

2013

• ORNL VFP 2013 summer program application Production runs on OLCF ‘Titan’ (GPUs)

Recent Proposal Submissions:

• NSF REU for computational science program with HPC:

Student stipends in STEM; 100+ cores buy-in at FSU HPC (Sep 2012)

• NSF ‘Physics at the Information Frontier’: computational research proposal (Nov 2012)

• Silicon Mechanics Inc application for research cluster (11 nodes, 176 compute cores, 2 GPUs, Feb 2013)

Computational Resources Available for Multi-Core Processing

Available computer labs at Florida A&M University with high-speed 1GB ethernet access to Florida State University High-Performance Computing Center via Florida Lambda Rail Network:

Department of Physics, Florida A&M University, Fred-Humphries Science Research Center

1530 S Martin-Luther-King Jr Blvd., Tallahassee, FL 32310:

Room 407:

1 computer lab with 8 desktop computers (8 processors each), 16 seat capacity

Room 407A:

1 computer lab with 5 desktop computers (8 processors each), 5 seat capacity

Room 418:

1 study room with 4 desktop computers (4 processors, 8 processors), 5 seat capacity

Room 420:

1 Apple MAC mini-cluster (20 cores), PI: R O’Neal (Physics)

Room 207 (Jones Hall):

1 computer lab with 10 desktop computers (dual-core machines), 10 seat capacity

Department of Chemistry, Fred-Humphries Science Research Center, 1530 S Martin Luther King Jr Blvd, Tallahassee, FL 32310:

Room 116:

1 computer lab with 20 desktop computers (dual-core machines), 40 seat capacity (shared with Environmental Science Institute)

Department of Computer and Information Sciences, Benjamin-Banneker Technical Bldg A, 1333 Wahnish Way, Tallahassee, Florida 32307:

Basement:

1 computer lab with 40 desktop computers (dual-core machines), 40 seat capacity

1 computer lab with 10 desktop computers (dual-core machines), 10 seat capacity

1 GPGPU computer cluster, 20 cores

FAMU-FSU COLLEGE OF ENGINEERING, Department of Electrical and Computer Engineering,

2525 Pottsdamer Street, Tallahassee FL 32310-6046:

Cluster with 100 Barcelona 8-core processors

Major Equipment Available to FAMU-FSU School of Engineering:

High-Performance Computing Center, The Florida State University, Department of Scientific Computing, Dirac Science Library, Tallahassee, Florida 32306-4120:

Computing:

1 Admin Node [Dell PowerEdge 2970 server]

Dual, Quad-Core Opteron 2356, 2.3 GHz, 16 GB RAM

1 Virtualization Node for services like Moab and Torque [Dell PowerEdge R905 server]

Quad, Quad-Core Opteron 8356, 2.3 GHz, 32 GB RAM

3 Login nodes, [Dell PowerEdge 2970 severs]

Dual, Quad-Core 2387 2.8 GHz Opterons, 16 GB ram

3 Login nodes [Dell PowerEdge 2970 servers]

Dual, Quad-Core 2382, 2.6 GHz, 16 and 32 GB ram

4 Login nodes [Dell PowerEdge 2970 servers]

Dual, Quad-Core 2356, 2.3 GHz, 16 GB ram

22 Compute nodes [Dell PowerEdge R815 servers]

Quad, 48-core 6174 2.2 GHz Opterons, 128 GB total (2.7 GB per core)

Total of 1056 cores

Mellanox ConnectX IB Cards

128 Compute Nodes [Dell PowerEdge SC 1435 Rev B servers]

Dual, Quad-Core 2356 2.3 GHz Opteron, 2 GB RAM per core

Total of 1024 Cores

Mellanox ConnectX IB Cards

128 Compute Nodes [Dell PowerEdge SC 1435 Rev B servers]

Dual, Quad-Core 2382 2.6 GHz Opteron, 2 GB RAM per core

Total of 1024 Cores

Mellanox ConnectX IB Cards

128 Compute nodes [Dell PowerEdge SC1435 servers]

Dual, Dual-Core 2220 2.8 GHz Opterons, 2 GB RAM per core

Total of 512 Cores

Mellanox Cheetah IB Cards

1 Blade Chassis [Dell PowerEdge M1000e ]

12-M605 Blades with 2, Quad-Core 2382 2.6 GHz Opteron, 2 GB RAM per core

2-M905 Blades with 4, Quad-Core 8382 2.6 GHz Opteron, 4 GB RAM per core

Total of 128 Cores

Mellanox DDR IB Cards

1 3Leaf SMP system

12 compute nodes which can be configured into 1 to 12 virtual SMP machines

Total of 576 GB of memory and 144 cores

The system has a dedicated QDR switch and each node utilizes a dual QDR connection

Storage:

• Panasas Parallel Storage: 12 Shelves of 190 TB of Panasas Parallel OSD Storage, 156 TB usable

in a RAID-6 like configuration;

• Lustre Secondary Storage

Networking:

Eight Stacked Dell PowerConnect 6248 Switches for Administrative Network (IP), CISCO 6509e Chassis for Storage Fabric (IP), CISCO 6509e Chassis for Storage Fabirc and WAN access via Dual redundant 10 GBE for Uplink to campus, QLogic QDR 12300 switch, CISCO 7024 Infiniband 288-port Chassis, Qlogic QDR 12300 (1) and 12200 (5) stack for the the Dual-Core 2220 2.8 GHz Opterons nodes