Testing a simulator¶
OpenFLUID < 2.2.0¶
Creating tests¶
Create a tests
folder at the root of the simulator.
In the tests folder, add .IN
and .REF
directories:
.IN
directories are OpenFLUID datasets..REF
directories are OpenFLUID reference outputs that are expected to be generated by the dataset. Example:
SimulatorA
├─ CMake.in.config
├─ CMakeLists.txt
├─ SimulatorA.cpp
├─ wareshub.json
└─ tests
├─ test1.IN
├─ test1.REF
├─ test2.IN
├─ test2.REF
└─ test3.IN
Running tests¶
In order to run simulations with .IN
datasets inside tests folder, use this OpenFLUID variable inside CMake.in.config
file.
Tests names must be datasets placed in the tests
subdirectory. Each dataset in the subdirectory must be names using the test name and suffixed by .IN
SET(SIM_TESTS_DATASETS test1 test2)
After configuration and build, running ctest
command in the build directory will run a simulation for each datasets defined in SIM_TESTS_DATASETS
variable.
The test passes if the simulation runs to the end.
A simulation test output is located in the build directory at path tests-output/<test-name>.OUT
.
At this step, no reference comparison is made. A comparison can be done manually.
Advanced tests with outputs comparison¶
In order to make an automatic comparison as a test, copy this block inside simulator CMakeLists.txt
file after OPENFLUID_ADD_SIMULATOR(SIM)
line.
This block creates new tests that compare outputs with references. If no reference is found, comparison is not made.
It requires the installation of the
OpenFLUID output diff script (python
and pip
required).
# [SKIP-2.2]>-------
FIND_PACKAGE(Python COMPONENTS Interpreter REQUIRED)
IF(Python_FOUND)
MESSAGE(STATUS "Enable comparing tests")
FOREACH(DATASET ${SIM_TESTS_DATASETS})
SET(TESTS_OUTPUT "${CMAKE_BINARY_DIR}/tests-output")
SET(REF_DIR "${CMAKE_SOURCE_DIR}/tests/${DATASET}.REF")
IF(IS_DIRECTORY ${REF_DIR})
SET(OUT_DIR "${TESTS_OUTPUT}/${DATASET}.OUT")
SET(REPORT_DIR "${CMAKE_BINARY_DIR}/tests-output/${DATASET}.REPORT")
ADD_TEST(NAME ${SIM_ID}-${DATASET}-diff
COMMAND "of-diff" "${REF_DIR}" "${OUT_DIR}" "${REPORT_DIR}" "--omit-na")
ELSE()
MESSAGE(WARNING "No reference found for dataset ${DATASET}")
ENDIF()
ENDFOREACH()
ENDIF()
# -------<[SKIP-2.2]
For more information, check the script documentation. Feel free to change comparison arguments if needed.