by Roger Rintala | June 11, 2015 12:30 pm
Work presented at the AHS 70th Annual Forum demonstrates that extracts are invaluable for both data reduction and quantitative analysis.
In their paper, “Turbulence Transport Phenomena in the Wakes of Wind Turbines”, Jha et al, show that data reduced by three orders of magnitude still retains full fidelity enabling quantitative analysis not possible before.
See the movie created for this project
Review the paper
Source URL: http://blog.ilight.com/ahsforum70/
by Roger Rintala | June 4, 2015 4:17 pm
Intelligent Light and Georgia Tech Researchers Achieving Breakthrough CFD Scalability at 64,000 cores
Research Leading Toward Practical Extreme Scale CFD
Rutherford, NJ – June 4, 2015
Intelligent Light, in collaboration with scalable solver developers at Georgia Tech and HPC experts at Lawrence Berkeley National Laboratory, is achieving breakthrough CFD scalability running the AVF-Leslie combustion simulation code on up to 64,000 cores on supercomputers at the Department of Energy’s National Energy Research Scientific Computing Center. The project is bringing together Intelligent Light’s expertise in computer science, software and hardware architecture, solver integration, data management, and the practical application of CFD to deliver scalable analysis methods and in situ infrastructure for extreme scale knowledge discovery.
High performance computing (HPC) is a necessity for the pursuit of high-fidelity, time-accurate simulations of sophisticated physics. HPC makes it possible to run these highly detailed simulations and deliver results in reasonable timeframes. When running simulations using thousands of cores however, the time to write, re-read and post-process the resulting files using traditional volume-based post-processing is impractical or impossible. When results are not reviewed and desired simulation runs not performed due to these limitations, the cost is wasted computing resources and lost science. In situ methods enable analysis of full spatiotemporal resolution data while it is still resident in memory, thereby avoiding the costs associated with writing very large data files to persistent storage for subsequent, post hoc analysis.
Scalable CFD analysis with extreme scale computing
For the AVF-Leslie code, a derivative of Georgia Tech’s Leslie3D solver code, breakthrough scalability is being achieved when running on up to 64,000 cores and the code has been instrumented with VisIt/Libsim to enable in situ extraction of surfaces of interest. Previously, the code has been used for combustion simulations running up to 5,000 cores. The extracted surfaces are output to compact XDB files for secondary processing using FieldView, Intelligent Light’s highly efficient post-processing tool that is a mainstay of CFD analysis. XDBs retain full numerical fidelity, enable both automated report generation and interactive exploration and can be used for archives. Already, combustion researchers are simulating at this extreme scale and learning about phenomena never before possible to explore either numerically or experimentally.
“Intelligent Light has been known for years for our post-processing and visualization technology. As the computing landscapes shifts to high-performance clusters, integrating post-processing with CFD solvers presents an opportunity to create a truly scalable workflow,” said Steve M. Legensky, General Manager and Founder at Intelligent Light. “Although we have run the VisIt code at up to 98,000 cores on the LLNL BlueGene/Q systems, we are seeing the challenges that arise when integrating with a sophisticated physics code like AVF-Leslie. This project, as well as others we have executed for DOE and DoD are helping us to understand the issues that affect both post-processing and solver code performance so that we can help our customers be successful in the HPC world.”
DOE taps Intelligent Light expertise in pursuit of extreme scale coherency and production quality software for real-world science
The Department of Energy (DOE) selected Intelligent Light as part of a team led by Lawrence Berkeley National Laboratory — a team that also includes Kitware, Georgia Tech and Argonne National Laboratory — to address the challenges of extreme-scale computing and integrating CFD solvers with in situ methods. Software tools for integrating solvers with in situ methods at extreme scale must maintain coherency across tens to hundreds of thousands of processor cores and be production quality to produce useful scientific results.
“Today we see widening gaps between compute performance and I/O capability and in situ analysis is a key part of the solution. As we move toward the exascale regime, we will see 3 orders of magnitude increase in FLOPs performance while at the same time seeing only 2-3 times more I/O performance,” says Wes Bethel, Senior Computer Scientist at Lawrence Berkeley National Laboratory. “Next generation workflows must address this discrepancy while delivering ultra-scalable performance for applications and Intelligent Light is among the organizations that are developing the proven, production quality software that will be required to produce successful science from these machines.”
DOE has assembled an exclusive team to develop the next generation methods and tools for in situ workflows to be used in a wide range of HPC-based scientific applications. Libsim is a key interface for in situ applications. As Libsim is tightly coupled to the solver, Intelligent Light is working with solvers to integrate this interface. Intelligent Light is a leading developer and maintainer of the open-source VisIt application and Libsim, both developed by the DOE.
XDB – Extract database files provide essential capability for interrogation
The use of extracts permits post hoc interactive exploration using standard tools without requiring the user to know what they want to see in advance. XDBs files are 10-1000 times smaller than solution files and are computationally efficient to create and save. CFD users can utilize automation to analyze large volumes of data, apply new generation techniques to identify important features from across vast datasets, and maintain the ability to explore solutions interactively using FieldView – the highly efficient, user-centric post-processing product long a favorite tool of CFD practitioners across industry and research around the world.
Leading the way forward with HPC
In working on research and production projects at the extreme scale, Intelligent Light is developing leading edge expertise and experience solving the challenges that occur when in situ methods are deployed in real world applications at the extreme scale. By understanding and solving the scalability and workflow issues at 64,000 cores and beyond, the use of HPC and in situ will be accelerated for all Intelligent Light customers.
To support the development and deployment of in situ methods, an SC15 workshop has been organized. Research results are being periodically presented as Intelligent Light’s research and development progress. The combustion study results were recently presented at the 27th International Conference on Parallel Computational Fluid Dynamics and presentations on related research are planned for AIAA SciTech in January, 2016.
This work is supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research under Award Number DE-SC0012449.
About Intelligent Light
Winners of multiple IDC HPC Innovation Excellence Awards, Intelligent Light provides industry-leading software and services that unlock the power and value of a highly productive CFD workflow for engineering and research organizations in a variety of industries around the world. The company’s flagship FieldView™ product line is the most widely used CFD post-processing software for engineering and research, encompassing data management, workflow automation, visualization, and more. Intelligent Light’s expert staff provides production-related engineering services, while its Applied Research Group conducts pure research on the cutting edge of CFD science. With customer success its paramount goal, Intelligent Light is driving real-world solutions to the toughest challenges in CFD today. www.ilight.com
About Berkeley Lab
Lawrence Berkeley National Laboratory addresses the world’s most urgent scientific challenges by advancing sustainable energy, protecting human health, creating new materials, and revealing the origin and fate of the universe. The Berkeley Lab Computing Sciences organization provides the computing and networking resources and expertise critical to advancing the Department of Energy’s research missions: developing new energy sources, improving energy efficiency, developing new materials and increasing our understanding of ourselves, our world and our universe. Founded in 1931, Berkeley Lab’s scientific expertise has been recognized with 13 Nobel prizes. The University of California manages Berkeley Lab for the DOE’s Office of Science. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time.
Source URL: http://blog.ilight.com/64000-cores_in-situ-cfd/
by Roger Rintala | April 9, 2015 6:54 am
The accessibility of HPC via cloud computing offers tremendous flexibility for CFD users with peak workload demands as well as for organizations and consultants who do not maintain HPC systems in house.
By designing a CFD workflow that maximizes the use of HPC systems and eliminates the transfer of volume data sets, productivity gains can be tremendous. The ability to run high resolution, time dependent simulations and full suites of design points allow every idea to be thoroughly vetted. Intelligent Light sponsored research used this approach to help a single researcher perform over 60 simulations and evaluate nearly 3TB of data for the AIAA High Lift Prediction Workshop. Result files were post-processed remotely and only compact XDB files were transferred to the user’s local workstation.
Learn how this was accomplished and see how this approach can make your CFD workflow more capable and productive.
Source URL: http://blog.ilight.com/hpccloudcfdxdbview/
by Roger Rintala | April 1, 2015 6:58 pm
Big Data, Big Challenge
Capturing helicopter interactions with a ship’s airwake in flight simulation made possible with in-situ XDB workflow.
The CREATE™–AV team undertook a simulation challenge last year: coupling the CFD solver HPCMP CREATE–AV Kestrel to the Navy flight simulator CASTLE® for a fully coupled rotorcraft simulation in a ship’s airwake. While high-fidelity ship airwake simulation data has been used with helicopter landing simulations, it had been based on queries of a static database. This project marks the first time that the flight simulation system has incorporated helicopter and ship airwake interactions into the simulation. Capturing the important interactions results in much higher realism and training quality.
Intelligent Light provided software and expertise to solve extraordinary challenges presented by the size of the data, the need to move it quickly, and to process and visulize the results. Intractable with traditional CFD workflows, Intelligent Light’s unique In-situ XDB workflow solved the problem.
The approach, hover and landing sequence had a simulation duration of 45 seconds. The time to save, transfer and read these visualization files along with the massive space required to save them emerged as a significant roadblock to success.
In-situ XDB Workflow via VisIt/libsim provided the solution. The CREATE–AV team, aided by Intelligent Light, integrated VisIt’s in-situ post-processing library libsim into Kestrel. The resulting workflow created and saved FieldView XDBs (extract databases) as the simulation ran on the HPC resource. FieldView was then used for all visual flight checks and movie generation, without the burden of dealing with the volume grid and results.
“The long time-scales and high frequency content due to the blade motion made visualization a challenge, so an ability to create feature extracts was created using VisIt libsim’s capability to extract FieldView13 xdb files. This enabled the creation of animations at a physical resolution of 60Hz (every five iterations) over the 45 seconds simulated. The resulting 2700 extracts were post-processed using FieldView in batch and in parallel to render the animations in a few hours rather than the days it would have otherwise taken. This capability should be in a future release of Kestrel.”
- J. Forsythe, C. Lynch, S. Polsky, and P. Spalart, “Coupled Flight Simulator and CFD Calculations of Ship Airwake using HPCMP CREATE™-AV Kestrel“,
AIAA paper 2015-0556
See AIAA 2015-0556 for details and acknowledgements.
Source URL: http://blog.ilight.com/in-situ-xdb-workflow-allows-coupling-of-cfd-to-flight-simulator-for-ship-airwakehelicopter-interaction/
by Roger Rintala | February 17, 2015 3:39 am
Visualizing Success: SBIRs Move Software into the Field
Read the DOE Case Study 
Intelligent Light is using SBIR grants to tap the
power of VisIt, a visualization program supported in
part by the Advanced Scientific Computing Research (ASCR) program in the Department of Energy (DOE) Office of Science.
Intelligent Light was selected as a trusted vendor who could provide the expertise and discipline to tailor tools to industrial workflows while commercializing and hardening the government developed code. Intelligent Light is providing industrial grade software development, engineering service and customer support to end users.
The project enables Intelligent Light to hold down development cost and end-user investments by utilizing open source code developed by DOE. Moreover, the project is helping Intelligent Light customers move to in-situ post-processing years earlier than if IL had developed all of the required technology from scratch.
Eric Brugger and other Lawrence Livermore National Laboratory scientists designed VisIt to visualize the mounds of simulation data DOE supercomputers generate. It’s built to scale well and it’s well suited for in situ processing.
“VisIt … is an open-source code, free to download and modify, as thousands of researchers have. With no licensing fees, merging VisIt into FieldView will hold
down costs for small- and medium-sized companies using it on HPC systems.”
Intelligent Light responded to a call by ASCR to harden and commercialize DOE-supported software for high performance computing applications, making the tools robust and accessible. Working under a Phase II grant of nearly $1 million, Intelligent light is integrating VisIt into our FieldView HPC suite. The funding is from a U.S. Small Business
Innovation Research (SBIR) program, which provides capital for early research and development with commercial potential.
VisIt’s big-data capacity allows FieldView to handle a wider range of CFD visualization tasks. “But ‘if you want to turn it into a commercial product, you really have to spend the time to customize things to people’s workflows,’ Brugger says. ‘We’ve customized it to our workflows in the DOE community,’ but it doesn’t always fit other users workflows.”
ASCR recognized it needs help to move VisIt and other DOE-supported solutions for high-performance computing into commercial use. Through the SBIR program, it called
for small businesses able to take DOE codes and “shrink wrap” them into accessible tools.
The collaboration “shows DOE is taking its research and development and spawning new technology – which is one of our missions,” says Benjamin Grover, division leader for
Applications, Simulations, and Quality at Livermore.
With a Phase II grant of nearly $1 million, Intelligent Light is integrating VisIt into its FieldView HPC suite. Later this year, customers will be able to choose either the standard FieldView code or VisIt to visualize CFD data.
Read the DOE Case Study 
Source URL: http://blog.ilight.com/visualizingsuccess/
Copyright ©2015 Intelligent Light and FieldView unless otherwise noted.