Testing a simulator

OpenFLUID < 2.2.0

Creating tests

Create a tests folder at the root of the simulator. In the tests folder, add .IN and .REF directories:

  • .IN directories are OpenFLUID datasets.
  • .REF directories are OpenFLUID reference outputs that are expected to be generated by the dataset. Example:
SimulatorA
├─ CMake.in.config
├─ CMakeLists.txt
├─ SimulatorA.cpp
├─ wareshub.json
└─ tests
   ├─ test1.IN
   ├─ test1.REF
   ├─ test2.IN
   ├─ test2.REF
   └─ test3.IN

Running tests

In order to run simulations with .IN datasets inside tests folder, use this OpenFLUID variable inside CMake.in.config file.
Tests names must be datasets placed in the tests subdirectory. Each dataset in the subdirectory must be names using the test name and suffixed by .IN

SET(SIM_TESTS_DATASETS test1 test2)

After configuration and build, running ctest command in the build directory will run a simulation for each datasets defined in SIM_TESTS_DATASETS variable. The test passes if the simulation runs to the end.
A simulation test output is located in the build directory at path tests-output/<test-name>.OUT. At this step, no reference comparison is made. A comparison can be done manually.

Advanced tests with outputs comparison

In order to make an automatic comparison as a test, copy this block inside simulator CMakeLists.txt file after OPENFLUID_ADD_SIMULATOR(SIM) line.
This block creates new tests that compare outputs with references. If no reference is found, comparison is not made.
It requires the installation of the OpenFLUID output diff script (python and pip required).

# [SKIP-2.2]>-------
FIND_PACKAGE(Python COMPONENTS Interpreter REQUIRED)
IF(Python_FOUND)
  MESSAGE(STATUS "Enable comparing tests")
  FOREACH(DATASET ${SIM_TESTS_DATASETS})
    SET(TESTS_OUTPUT "${CMAKE_BINARY_DIR}/tests-output")
    SET(REF_DIR "${CMAKE_SOURCE_DIR}/tests/${DATASET}.REF")
    IF(IS_DIRECTORY ${REF_DIR})
      SET(OUT_DIR "${TESTS_OUTPUT}/${DATASET}.OUT")
      SET(REPORT_DIR "${CMAKE_BINARY_DIR}/tests-output/${DATASET}.REPORT")
      ADD_TEST(NAME ${SIM_ID}-${DATASET}-diff
              COMMAND "of-diff" "${REF_DIR}" "${OUT_DIR}" "${REPORT_DIR}" "--omit-na")
    ELSE()
      MESSAGE(WARNING "No reference found for dataset ${DATASET}")
    ENDIF()
  ENDFOREACH()
ENDIF()
# -------<[SKIP-2.2]

For more information, check the script documentation. Feel free to change comparison arguments if needed.