Menu

Success Story: Enabling parallel mesh adaptation with Treeadapt

Success story # Highlights:

  • Keywords:
    • CFD
    • Mesh adaptation
    • Parallel performance
    • Load balancing
  • Industry sector: Aeronautics, High Performance Computing
  • Key codes used: AVBP, Treepart, Treeadapt
Mesh adaptation of an inviscid eulerian vortex inital mesh (left), First adapted grid (middle), Final mesh (left)

Organisations & Codes Involved:

cerfacs

CERFACS is a theoretical and applied research centre, specialised in modelling and numerical simulation. Through its facilities and expertise in High Performance Computing (HPC), CERFACS deals with major scientific and technical research problems of public and industrial interest.

avbp

AVBP is a parallel Computational Fluid Dynamics (CFD) code that solves the three-dimensional compressible Navier-Stokes on unstructured and hybrid grids. Its main area of application is the modelling of unsteady reacting flows in combustor configurations and turbomachines. The prediction of unsteady turbulent flows is based on the Large Eddy Simulation (LES) approach that has emerged as a prospective technique for problems associated with time dependent phenomena and coherent eddy structures.

scientific Challenge:

CFD remains highly dependent on mesh quality. Advanced meshing software is generally limited to sequential or shared memory architectures. Therefore, meshing requires tens of hours to generate highly complex grids and it is highly dependent on user experience. Refinement zones are also bounded by standard geometrical shapes. To bypass these bottlenecks, codes have turned to mesh adaptation as a solution but massively parallel mesh adaptation workflows remain scarce and require efficient load balancing, interpolation and remeshing techniques.

Solution:

Building on previous experience with the INRIA MMG library [1] and a partnership with the EPEEC project [2] during which an efficient mesh partitioning and load balancing library called Treepart [3] was developed, EXCELLERAT has developed a new application/library called Treeadapt for massively parallel mesh adaptation. Currently operating in full tetrahedral grids, Treeadapt generates a first partitioned domain where MMG can be called independently while freezing the parallel interfaces. A rebalancing and adaptation iterative process then occurs until the whole domain is within a user provided tolerance with regards to the error estimator (example: gradient of density, Hessian of the velocity).  

Scientific impact of this result:

So far, simulations using AVBP relied on static commercial software generated meshes. This bounded the quality of results to the experience and insight of the user performing the simulation. Furthermore, these mesh generation tools are mostly non-parallel and require days to mesh complex cases.

Recent work on efficient load balancing within the EPEEC project allowed CERFACS to use the open source library Treepart to create a new tool/library called Treeadapt that allows massively parallel mesh adaptation. Treeadapt uses its mesh partitioning and load balancing algorithms based on the ZOLTAN library [4] to efficiently decompose the domains taking into account the architecture intricacies of the system it is running on. These load balancing algorithms coupled with MMG allow faster and more efficient remeshing.

Then, Treeadapt has been used to improve the simulation of a 42 injector rocket engine demonstrator simulation called BKD [5] in the PRACE project: the prediction of combustion instabilities in liquid rocket engines (RockDyn). This simulation in collaboration with EM2C Centrale Supelec (T. Schmitt) reaching one Billion tetrahedras in less than 30 minutes, using 4,096 AMD Epyc 2 cores (compared to 70 hours with a standard meshing tool).

Benefits for further research:

  • From three days to one hour to generate a one Billion element mesh
  • Reduce result quality dependency on user experience
  • Improve high fidelity results
Refinement detail of one injector of the BKD use case.

Unique Value:

AVBP offers the possibility to use unstructured grids than can be automatically adapted to the flow at runtime allowing for automatic tracking of the zones of interest.  Additionally mesh quality criteria in AMR can compensate for any defects on the original grid that might lead to numerical issues.

Products/services:

  • Tools : Treeadapt is part of an opensource package. Users should provide CAD software/mesher.
  • Training about Enabling parallel mesh adaptation with Treeadapt
  • Consultancy about mesh adaptation methods applied to aeronautics and combustion
  • Best practise guide about Enabling parallel mesh adaptation with Treeadapt

References:

  1. [1] https://www.mmgtools.org
  2. [2] https://epeec-project.eu
  3. [3] https://arxiv.org/pdf/2008.00832.pdf
  4. [4] https://cs.sandia.gov/Zoltan
  5. [5] S. Gröning et al, Journal of Propulsion and Power 32(3), 560 (2016)

 

This work benefited from private collaboration with INRIA and CORIA-CNRS as well as with the EPEEC project (grant agreement No 80105).

If you have any questions related to this success story, please register on our Service Portal and send a request on the “my projects” page.