Intelligent End-to-End Petascale Framework

I have always felt a strong bond with the sea, and that shaped my approach to science and business, enabling me to work on a rich variety of topics and domains.

Surprisingly, there exists a close analogy between surfing and paradigm-shifting events in science and business. The following figure from barefootsurftravel.com (How to Find & Catch Unbroken Waves)is useful in making this connection.


To surf a wave, you need the right conditions in the ocean to create surfable waves. And then you need to possess an instinct about which wave to catch, and when to catch it. It can mean the difference between being at the bleeding edge, where the technology is at high risk of being unreliable and has not gained traction, or the cutting edge, where the technology is past the proof point and gaining traction.

If you catch a wave too early when it is only a bump, then you will not be able to ride it. If you catch a wave too late when it is breaking or has already broken, then not only will you not be able to ride it, but you risk getting hurt.
 
Similarly, most of the time spent developing science and business involves incremental progress. But every so often, there is a major shift or breakthrough that is a game changer. The trick is to anticipate or notice the upwelling of such shifts, be prepared to ride it at the proper time, and to know when to get out.

I have been lucky enough to anticipate, catch and ride several paradigm-shifts in my career. I have also had some misses where my timing was off, and I ended up at the bleeding edge. An example is development of applications for electronic medical records and online transactions from the physician to pharmacy. Although common place now, they were still too new for the marketplace to adopt in the 1990s. Luckily the number of my successes have outnumbered the misses by a large margin.

A successful example is my anticipation in 2005 that peta-scale simulations would become mission critical back, which led me to start developing an end-to-end framework for simulations to be run on massively parallel computers that were yet to be built. Figure 2 below shows a plot of the largest number of particles possible in a simulation over the years. The number of particles in a kinetic simulation is a good proxy for the size of the simulations that are possible. I was directly involved in marching the state-of-the-art in simulations along the trajectory shown.

I put together a multidisciplinary team of computational physicists, computer scientists, and scientific visualization experts, and we started adapting our simulation algorithms to run efficiently on massively parallel computers. Data processing and visualization were other technical challenges that needed to be overcome. Knowledge discovery from large simulations remains a major challenge and is becoming more urgent as the march towards ultra-scale computing with millions of cores continues. I was particularly dismayed by the lack of useful scientific visualization techniques, especially in the domain of plasma physics, which was my primary area of interest at the time. I hired on Burlen Loring, a talented visualization expert, and we ended up having a very fruitful collaboration.


There are at least three attendant issues in scientific visualization. First, the traditional paradigm of running the simulations and saving the data to disk for post-processing creates an issue in that it is only feasible to save the data at a small number of time slices. This low temporal resolution of the saved data is a serious handicap in many studies where the time evolution of the system is of principle interest. Second, the simulation data can be quite noisy and finding the events of interest in the noise and tracking them over time is challenging. Finally, off-the-shelf visualization packages are severely inadequate for most applications and significant customization and even development of new techniques are often required. Without close collaboration between scientists and experts in visualization software, it is very difficult for most end users to adapt these techniques for their specific use. This has been a major impediment to unraveling the true potential of scientific visualization as a knowledge discovery tool.

I took a methodical approach to addressing these issues. To address the first issue, we developed in-situ visualization strategies. The idea is to minimize data storage by extracting important features of the data and saving features, rather than raw data, at high temporal resolution. Technical details can be found in H. Karimabadi, P. O’Leary, B. Loring, A. Majumdar, M. Tatineni, and B. Geveci, “In-situ visualization for global hybrid simulations,” in the Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery (XSEDE ’13). Association for Computing Machinery, July 2013, 8-16.

Figure 3 below shows a snapshot of a 3D kinetic simulation of the solar wind interacting with the Earth’s magnetic field.  The left panel shows a 2D slice of density in the equatorial plane. The right panel shows the streamlines in the equatorial plane in the zoomed in area marked by the white rectangle. The color is based on the magnitude of the ion flow speed. Only a segment of the 3D simulation box is shown. This diagnostic enabled us to demonstrate the formation of large flow vortices on the flanks of the magnetosphere.

We addressed the second problem by implementing a variety of detection and tracking techniques. My particular insight here was to introduce techniques from machine learning and computer vision into scientific visualization.

Figure 4 shows an example of the utility of our approach. The physics problem that we were trying to solve was to use our first-of-its-kind 3D kinetic simulations to detect and track the formation of the so-called flux transfer events (FTEs) in the Earth’s magnetosphere. FTEs are generated due to magnetic reconnection process which is the primary driver of space weather. FTEs have been observed in spacecraft data, but many details such as their shape and evolution have been subject of much controversy. Plotting the magnetic field lines in the simulation (the top panel) is not informative and the FTEs are very hard to spot. The bottom panels show the same data but processed through our visualization technique. It detects and tracks the FTEs and provides their signature at different orientations in 3D and along different 2D cuts. This visualization provided the most detailed view of FTE formation and evolutionary dynamics in 3D global kinetic simulations.

Finally, we developed custom visualization techniques, that in turn facilitated major advances in the field of plasma turbulence (Karimabadi et al., Coherent structures, intermittent turbulence, and dissipation in high-temperature plasmas Physics of Plasmas 20, 012303 (2013); https://doi.org/10.1063/1.4773205) as well as space weather (Karimabadi et al., The link between shocks, turbulence, and magnetic reconnection in collisionless plasmas Physics of Plasmas 21, 062308 (2014); https://doi.org/10.1063/1.4882875). Here are some illustrative examples from these publications.
 
One of the fundamental problems in turbulence is the mechanisms for dissipation. Using the largest peta-scale kinetic simulation at the time, we extended the resolution of the simulation down to electron scale. Our work revealed a number of surprises and has since become one of the key papers in the field (Karimabadi et al., Coherent structures, intermittent turbulence, and dissipation in high-temperature plasmas Physics of Plasmas 20, 012303 (2013); https://doi.org/10.1063/1.4773205). In particular, it revealed that formation and subsequent reconnection of current sheets with electron scale thickness is a dominant source of dissipation in turbulence. To demonstrate this point, we introduced and adapted the line integral convolution (LIC) technique for plasma physics. Figure 5 shows, using LIC, the time evolution of the magnetic field lines colored by the magnitude of the field. A movie of the generation and subsequent evolution of the current density further supported the importance of current sheet formation in turbulence (see below).

In a second example, we used state-of-the-art global kinetic simulations along with our custom visualization techniques (e.g., LIC, streaklines, pathlines) to explore the dynamics of the Earth’s magnetosphere under different solar wind conditions Karimabadi et al., The link between shocks, turbulence, and magnetic reconnection in collisionless plasmas Physics of Plasmas 21, 062308 (2014); https://doi.org/10.1063/1.4882875). We were able to demonstrate, for the first time, the formation of jets in the magnetosphere. This is shown in Figure 6 below. These jets are parcels of high velocity solar wind that are able to penetrate the Earth’s protective magnetic shield. In the left panel, we show the dynamical pressure in an area of the simulation zoomed in around the region of jet formation. Some of the jets reach the magnetopause, triggering space weather effects, while others terminate closer to the bow shock. The right panel shows the LIC of magnetic field colored by dynamical pressure for an area zoomed in around a strong jet marked by the bow wave in the left panel that has triggered a flux transfer event at the magnetopause. Also evident in the right panel is the formation of so-called plasmoids (loop structures in LIC) in the magnetosheath. This was the first simulation that demonstrated the formation of such structures in the magnetosheath.

Subsequent spacecraft observations from NASA’s latest mission, MMS, have verified these predictions. A movie of this jet formation is as follows.

Another surprise was that under certain conditions, large flow vortices can form in the magnetosheath. Figure 7 below shows the development of the vortex using streaklines technique. There is a downward flow (pink streamlines) that pushes the plasma out sunward. In this case, the sunward flow gets caught in the velocity shear in the magnetosheath and wraps up into a vortex.

There is a downward flow (pink streamlines) that pushes the plasma out sunward. In this case, the sunward flow gets caught in the velocity shear in the magnetosheath and wraps up into a vortex. A movie of this effect is shown here.

A third example is from the largest global kinetic simulation which we conducted in 2014. This run remains the largest simulation of its kind to date. Through custom visualization, we were able to show the formation of helical patterns in front of the Earth’s magnetosphere. These structures are due to pick-up and subsequent gyration of the ions reflected from the Earth’s bowshock as shown in this movie.

This movie was produced in collaboration with Burlen Loring, now at Berkeley Visualization Lab. Our simulation data was also used for a fulldome production called “Solar Superstorms”, which has been shown across the U.S. and can be viewed here: Cosmic Journey - Solar Superstorms