View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0003846OpenFOAMFeaturepublic2022-05-31 11:292022-05-31 14:10
Reporterdaronciyakian Assigned Tohenry  
PrioritynoneSeverityfeatureReproducibilityalways
Status closedResolutionno change required 
PlatformGNU/LinuxOSUbuntuOS Version18.04
Product Version9 
Summary0003846: dnsFoam parallelization problem
DescriptiondnsFoam doesn't work in parallel. It works very fine in serial but can not work parallel. I tried 3 different decomposition methods but output stay same.

decompose methods tried:
1)
numberOfSubdomains 8;

method hierarchical;

simpleCoeffs
{
    n (4 1 1);
}

hierarchicalCoeffs
{
    n (4 2 1);
    order xyz;
}

manualCoeffs
{
    dataFile "cellDecomposition";
}

2)
numberOfSubdomains 8;

method simple;

simpleCoeffs
{
    n (8 1 1);
}

hierarchicalCoeffs
{
    n (4 2 1);
    order xyz;
}

manualCoeffs
{
    dataFile "cellDecomposition";
}
3)
numberOfSubdomains 8;

method scotch;

simpleCoeffs
{
    n (8 1 1);
}

hierarchicalCoeffs
{
    n (4 2 1);
    order xyz;
}

manualCoeffs
{
    dataFile "cellDecomposition";
}


Error output:
[0]
[0]
[0] --> FOAM FATAL ERROR:
[0] calculated number of cells is incorrect
[0]
[0] From function Foam::Kmesh::Kmesh(const Foam::fvMesh&)
[0] in file Kmesh/Kmesh.C at line 76.
[0]
FOAM parallel run aborting
TagsNo tags attached.

Activities

henry

2022-05-31 11:34

manager   ~0012613

> dnsFoam doesn't work in parallel

That is correct, it relies on a Fourier transform for the UO forcing term and that has not been parallelised.

daronciyakian

2022-05-31 12:07

reporter   ~0012614

I can actually run 256^3 resolution in single core but I need to increase resolution to increase Re to keep reliability. 512 and 1024 is not possible with single core. Could you please give me some idea how can I make dnsFoam parallel. I mean how can I force the flow when we decompose flow field if it is necessary to use fft? or Is it possible to apply force without using fft and just introduce force in real space somehow? like Lundgren linear forcing in real space?

I read that
 "forcing performed in physical space, where all wavenumber modes are triggered as Lundgren proposes, distorts the redistribution of energy and hence the effects of particles on E(κ , t) cannot be quantified" https://doi.org/10.1063/1.4818553 page2


"the forcing of the first few low wavenumber modes in spectral space creates large fluctuations on the temporal evolution of the turbulence kinetic energy which can be of the same order of magnitude as the effect of the two-way coupling" https://doi.org/10.1063/1.4818553 page2

dnsFoam use spectral forcing scheme and If you apply force to small wavenumbers it is also claimed have some disadvantages. If you apply force to long range of wavenumbers you basically obtain Lundgren scheme. right?

This two scheme (spectral and real space schemes) have different backwards. Maybe my idea might be silly. why we can not apply this force in real space without using spectral wavenumbers?

henry

2022-05-31 12:56

manager   ~0012615

Last edited: 2022-05-31 12:57

 I wrote dnsFoam in ~1995 to test and compare LES models using the same numerics for both DNS and LES and it hasn't been used since. We receive no funding to maintain or further develop dnsFoam, in fact there has been no interest in it until your request.

Are you sure 2nd-order FV is appropriate for your DNS requirements? Most serious "box of turbulence" DNS studies use compact high-order FD, Fourier or spectral element methods.

daronciyakian

2022-05-31 13:56

reporter   ~0012616

Actually there is no completely perfect DNS. Spectral methods have drawback such as aliasing. If you are not sure about 2nd order FV for DNS you can't be sure your LES too. Up to filtering level you are using same DNS.

According to Moin and Mahesh "Finite difference schemes typically have lower levels of aliasing error than spectral methods. Aliasing error can cause either numerical instability or excessive turbulence decay .When central differences are used, the skew-symmetric form of the nonlinear terms ([u jui, j + (uiu j ), j]/2), is seen to have lower levels of aliasing error for both incompressible and compressible DNS. For finite-difference schemes, the combination of higher levels of differentiation error and lower levels of nonlinear truncation and aliasing error determines the overall error at the small scales. A rule of thumb is that second-order central difference schemes require about twice the resolution (in each direction) to achieve the same results as a spectral DNS" https://www.researchgate.net/publication/234038721_Direct_Numerical_Simulation_A_Tool_in_Turbulence_Research page 545-546

Thus, I thought that I can catch same accuracy by doubling resolution in each direction. I don't have any proof but I think dnsFoam more faster than spectral DNS. I haven't test any spectral DNS but I could do 128^3 resolution with single core in a few hours by dnsFoam. I guess it takes much more time in any typical spectral DNS. So doubling is not that much issue.

My final aim is also try to compare LES models with DNS. To be sure about consistency I would like to use OpenFOAM in both (DNS and LES).

So, I really need to solve forcing problem in parallel in dnsFoam.

henry

2022-05-31 14:10

manager   ~0012617

This is a feature request that would require significant funding.

Issue History

Date Modified Username Field Change
2022-05-31 11:29 daronciyakian New Issue
2022-05-31 11:34 henry Note Added: 0012613
2022-05-31 12:07 daronciyakian Note Added: 0012614
2022-05-31 12:56 henry Note Added: 0012615
2022-05-31 12:56 henry Priority immediate => none
2022-05-31 12:56 henry Severity block => feature
2022-05-31 12:57 henry Note Edited: 0012615
2022-05-31 13:56 daronciyakian Note Added: 0012616
2022-05-31 14:10 henry Assigned To => henry
2022-05-31 14:10 henry Status new => closed
2022-05-31 14:10 henry Resolution open => no change required
2022-05-31 14:10 henry Note Added: 0012617
2022-05-31 14:10 henry Category Bug => Feature