View Issue Details

IDProjectCategoryView StatusLast Update
0003597OpenFOAMContributionpublic2020-12-09 16:34
Reporterblttkgl Assigned Tohenry  
PrioritynormalSeverityfeatureReproducibilityN/A
Status closedResolutionsuspended 
PlatformGNU/LinuxOSOtherOS Version(please specify)
Summary0003597: A dynamic load balanced chemistry model for faster reactive simulations
DescriptionHey,

We have developed a chemistry model that mitigates the CPU load imbalance related to chemistry solution in parallel reactive simulations using dynamic load balancing. This is a known issue that severely limits the reacting flow simulation performance.

The loadBalancedChemistryModel does not require any third party dependency and currently runs perfectly well on OpenFOAM-7 and -8 releases. Our tests show us it can be easily added to -dev as well with minor tweaks.

There are couple of restrictions at the moment. First of all, it inherits directly from standardChemistryModel, so it can't be used with TDACChemistryModel at the moment. However, this extension is a very low hanging fruit and can easily be achieved. Second, it currently does not support surfaceArrhenius reactions, introduced in OpenFOAM-8. If our contribution is found interesting we can help to solve these issues.

I am adding the repository and the preprint of the implementation paper below.

Repository: https://github.com/blttkgl/DLBFoam
Paper: https://arxiv.org/abs/2011.07978

Best,

Bulut
TagsNo tags attached.

Activities

blttkgl

2020-11-23 10:35

reporter   ~0011733

I would also like to add that our model also includes a "reference mapping" model, which is a simple tabulation model that is somewhat similar to ISAT. It could be also contributed as a tabulation method alongside ISAT.

henry

2020-11-23 10:45

manager   ~0011734

Specific load balancing just for chemistry introduces a substantial maintenance overhead and only covers one requirement. If we added separate load balancing implementation for all distributed loads the code would become unmanageable. What we really need is a general load balancing implementation based on mesh redistribution which can handle mesh refinement/unrefinement, Lagrangian, film, chemistry etc. in which the distributed loads are supplied as weights to the balancing engine. Could you re-implement your load balancing in this more general form so that it can be used for these other very important purposes?

> "reference mapping" model, which is a simple tabulation model that is somewhat similar to ISAT

What is the benefit over ISAT? Is it always faster? Could in replace ISAT? If not when would you choose which method?

blttkgl

2020-11-23 11:10

reporter   ~0011735

Thanks for the reply.

We are aware of OpenFOAM's desire for a balancer that can be generalized based on cell distribution, and there are already some attempts for this in literature for instance for load-balanced AMR simulations in OpenFOAM. A re-implementation to do what you described would be a huge job with multiple people involved and is definitely out of our scope at the moment. Balancing the chemistry is simple, because chemistry is a cell-specific problem that does not require any neighbor or connectivity information, therefore it can easily be distributed-redistributed with minimum effort. Regarding maintenance of course I have to take your word for it, but during development we made sure to use existing methods and code as much as possible to minimize maintenance for us. It's also important to mention that as seen in the preprint in reactive simulations >95% of the time is spend on chemistry, so at least for reactive simulations balancing the chemistry is what really matters. In either case we think this is a useful tool for anyone out there wants to use OpenFOAM for reactive simulations and wanted to see if we can reach a greater number of people through official repository.

> What is the benefit over ISAT? Is it always faster? Could in replace ISAT? If not when would you choose which method?

Reference mapping method uses the mixture fraction (Z) field, which is defined as 0 at the pure oxidizer and 1 at the pure fuel conditions. For each iteration, it finds a cell satistfying the Z<tolerance_ criteria, solves it, and maps this solution to other subsequent cells satisfying the same condition. It does not replace ISAT, but offers an alternative especially for problems with clear "oxidizer" and "fuel" conditions like non-premixed combustion applications. The benefit over ISAT is it is a much simpler approach with only 1 tabulated solution as opposed to a binary tree full of solutions in ISAT. It is much less dependent to model constants, so it requires no "tuning" to get it working. We have been using this model in our publications using OpenFOAM for many years and decided to make it public together with the balancer. Related to above, just like ISAT this model does not solve the bottleneck issue in parallel simulations due to its process based implementation. The real speedup is obtained when its used together with the load balancing model we introduced.

henry

2020-11-23 11:38

manager   ~0011737

Last edited: 2020-11-23 11:50

Redistributing the chemistry differently to everything else will cause problems when general load-balancing and redistribution is implemented and given that local refinement and unrefinement is beneficial for many reacting problems redistribution based on the mesh changes AND chemistry is really needed. A starting point for this would be fvMeshDistribute which handles the bulk of the complexity.

It is not clear how much benefit the reference mapping approach has over ISAT, how much speedup do you get for say the tutorial cases? Have you tested it on the DLR_A_LTS case?

peltonp1

2020-11-23 12:29

reporter   ~0011740

About the generic load balancer:

I understand the need for a generic balancer and in fact it would be relatively easy to implement the current balancer to redistribute types T. However, I'm not sure how much of a performance gain there would be if you start tossing around objects that are inherently coupled. If a generic redistribution of load is truly needed, I would seriously consider switching from a "data parallel" to a "task based" parallelism throughout the code base and rely on a runtime scheduler (for example HPX). This would very likely be a major undertaking but would also fix many issues related to IO/AMR and allow for multiple levels of parallelism depending on the target architecture. I would personally be interested in taking part if such a project is ever started.

-Petteri

henry

2020-11-23 13:15

manager   ~0011741

Last edited: 2020-11-23 16:22

> However, I'm not sure how much of a performance gain there would be if you start tossing around objects that are inherently coupled.

All the functionality to do this is already present and is used effectively in snappyHexMesh. The load-balancing would not need to be done very frequently so the comms overhead would be small compared to the solution.

> If a generic redistribution of load is truly needed

Yes it is, particularly for mesh refinement/unrefinement and Lagrangian.

> "task based" parallelism

I do not think this would scale well to 1000s of cores but it would be interesting to see the results if you undertake the implementation.

blttkgl

2020-11-23 14:00

reporter   ~0011742

> Have you tested it on the DLR_A_LTS case?

No I haven't, but as I said the performance (and accuracy) of ISAT very much depends on model tolerances and constants. While ISAT might be performing faster compared to refmapping for the default model constants in the tutorial, the refmapping might be giving a more accurate results compared to a no-tabulation standard approach. I can try it out.

I understand from our conversation that Foundation is not particularly interested with our chemistry model. In that case you may close the issue.

Thank you for your time,

Bulut

henry

2020-11-23 14:25

manager   ~0011743

Last edited: 2020-11-23 17:35

The OpenFOAM Foundation and the OpenFOAM users are very interested in load-balancing in general, particularly for mesh refinement/unrefinement and Lagrangian; these have been the most requested but so far we have not secured funding to work on it. Load-balancing of chemistry would also be useful and would fit easily is the general framework based on fvMeshDistribute and I would expect this implementation to require less effort and additional code than implementing it from scratch specifically for chemistry. We have limited funding for code maintenance and need to be careful how much addition code is added for specific purposes when more general implementations are possible which reduce the maintenance load.

TDAC and ISAT are already a significant maintenance overhead and ISAT should be rewritten in a more general manner so it can be used for other purposes but we have no funding for this. The change to exact Jacobian evaluation also added a lot of code complexity making it harder to maintain and difficult to add new reactions and unfortunately did not provide the (or any) performance improvement that proposers of the change claimed. Adding an alternative to ISAT would be fine if there are clear advantages: if it were faster for a range of cases, simpler and could replace ISAT etc. Adding new code and the associated maintenance overhead has to be balanced carefully againt the need and advantage, particularly when no additional funding is provided for the maintenance.

If you can show for what range of cases refmapping is faster than ISAT for a range of cases we can consider including it in OpenFOAM and maintaining it. As refmapping is simpler than ISAT if it could replace ISAT altogether that would be even better.

henry

2020-12-09 16:34

manager   ~0011796

Pending a general implementation of load-balance as discussed.
Pending a detailed comparison between "reference mapping" and ISAT with the view to replacing ISAT if "reference mapping" proves more efficient.

Issue History

Date Modified Username Field Change
2020-11-23 10:27 blttkgl New Issue
2020-11-23 10:35 blttkgl Note Added: 0011733
2020-11-23 10:45 henry Note Added: 0011734
2020-11-23 11:10 blttkgl Note Added: 0011735
2020-11-23 11:38 henry Note Added: 0011737
2020-11-23 11:50 henry Note Edited: 0011737
2020-11-23 12:29 peltonp1 Note Added: 0011740
2020-11-23 13:15 henry Note Added: 0011741
2020-11-23 13:21 henry Note Edited: 0011741
2020-11-23 14:00 blttkgl Note Added: 0011742
2020-11-23 14:25 henry Note Added: 0011743
2020-11-23 14:25 henry Note Edited: 0011743
2020-11-23 14:26 henry Note Edited: 0011741
2020-11-23 16:22 henry Note Edited: 0011741
2020-11-23 17:35 henry Note Edited: 0011743
2020-12-09 16:34 henry Assigned To => henry
2020-12-09 16:34 henry Status new => closed
2020-12-09 16:34 henry Resolution open => suspended
2020-12-09 16:34 henry Note Added: 0011796