TDycore Development =================== This section contains topics that are useful to anyone developing the TDycore library or trying to use it in an application. Initializing the TDycore Library -------------------------------- Because the TDycore library uses MPI, PETSc, and other subsystems, we have defined a function to initialize these various subsystems at the beginning of any TDycore-based driver/program::: PetscErrorCode TDyInit(int argc, char *argv[]); Call this function where you would ordinarily call ``MPI_Init`` or ``PetscInitialize``. It has no effect on subsequent calls. You can check whether the library has been initialized with a call to:: PetscBool TDyInitialized(void); Similarly, we have defined a finalization function to be called at the end of a TDycore-based program::: PetscErrorCode TDyFinalize(void); Use this function instead of ``MPI_Finalize`` or ``PetscFinalize``. This ensures that all TDycore subsystems properly free their resources. Memory Allocation and Deallocation ---------------------------------- Within the TDycore library, the following functions are used to dynamically allocate and deallocate memory::: PetscErrorCode TDyAlloc(size_t size, void **result); PetscErrorCode TDyRealloc(size_t size, void **memory); PetscErrorCode TDyFree(void *memory); ``TDyAlloc`` allocates and zero-initializes a contiguous block of memory of the specified size in bytes, aligned to ``PETSC_MEMALIGN``. Use this function anywhere you would use ``malloc`` or any of the PETSc memory allocation functions like ``PetscMalloc``, ``PetscCalloc``, etc. ``TDyRealloc`` resizes an existing block of memory to a new size. Call this where you would call ``realloc`` or ``PetscRealloc``. ``TDyFree`` frees an allocated block of memory. Call this where you would call ``free`` or ``PetscFree``. Fortran 90 Interface ^^^^^^^^^^^^^^^^^^^^ We offer two equivalent subroutines for Fortran 90, similar to their PETSc counterparts::: TDyInit(ierr) TDyFinalize(ierr) Both accept an integer that stores an error code if these subroutines encounter an issue. Timers and Profiling -------------------- We use PETSc's `Logging machinery ` to understand the performance of the dycores. In particular, we've provided some high-level wrappers around the `PetscLogEvent ` object that make it very easy to add timers for functions and blocks of code. The TDycore Timers Subsystem ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The subsystem for timers and profiling is defined almost entirely in ``tdytimers.h``. When a dycore is initialized, a registry of timers is created when you call ``TDyInit()``. Once this is done, you can manipulate timers with functions and macros as described below. Make sure you include the ``tdytimers.h`` file wherever you use these timers. Function-Level Profiling ^^^^^^^^^^^^^^^^^^^^^^^^ The easiest way to use a timer is to instrument a function with a timer using ``TDY_START_FUNCTION_TIMER()`` and ``TDY_STOP_FUNCTION_TIMER()``. These macros automatically create/retrieve a timer for the function and start/stop it as you'd expect. You place the ``START`` macro at the top of a function, and the ``STOP`` one at the bottom. For example::: void do_some_expensive_things(TDy dy) { TDY_START_FUNCTION_TIMER() ... // Do the expensive things TDY_STOP_FUNCTION_TIMER() } There's no need to understand PETSc's logging objects--everything is done for you. These function-level timers are named after the functions in which they appear. Manually-Created Timers ^^^^^^^^^^^^^^^^^^^^^^^ Sometimes you want to time something that happens in the middle of a function. You can do this by calling the ``TDyGetTimer`` function with the ``TDyStartTimer`` and ``TDyStopTimer`` macros::: void do_various_things(TDy dy) { ... // Stuff happens here // Now we want to time a block of code in the middle of the function. PetscLogEvent timer = TDyGetTimer("important things"); TDyStartTimer(timer); ... // Important things happen here! TDyStopTimer(timer); ... // Other stuff happens here } This is a bit more involved--you need to know that timers are ``PetscLogEvent`` objects, for example, and you need to name your timers--but not too difficult. As you might have guessed, ``TDY_START_FUNCTION_TIMER`` and ``TDY_STOP_FUNCTION_TIMER`` are just wrappers around these constructs. ``TDyStartTimer`` and ``TDyStopTimer`` are themselves just macros that call ``PetscLogEventBegin`` and ``PetscLogEventEnd``, so you can always use those if you want more control. Profiling Stages ^^^^^^^^^^^^^^^^ PETSc allows an arbitrary number of logging/profiling "stages" to be defined so that you can organize your profiling into sections. These stages can be named for convenience. You can enter and exit a named stage with calls to ``TDyEnterProfilingStage(stageName)`` and ``TDyExitProfilingStage(stageName)``, where ``stageName`` is a string containing the name of the stage. TDycore provides these named stages, registering them in ``TDyInit``: * ``"TDycore Setup"``: for creating meshes, setting up initial conditions, calculating time-independent matrices and vectors, etc. * ``"TDycore Stepping"``, for timestepping * ``"TDycore I/O"``, for checkpointing, restarting, generating visualizations You can register your own named stages with ``TDyAddProfilingStage(stageName)``. All of this machinery is a thin wrapper around PETSc's ``PetscLogStage`` mechanism, which you can use if you prefer. The functions and macros above just simplify the bookkeeping. Generating Profiling Logs ^^^^^^^^^^^^^^^^^^^^^^^^^ Adding timers to a code is only part of profiling. You also need to generate profiling reports for runs of interest! Fortunately, this is easy--just add the ``-tdy_timers`` flag to your command line arguments to generate a performance log. This log is named ``tdycore_profile.csv``. It's a comma-separated variable file containing all performance data collected by PETSc. The timers you've created show up in the profile just like those embedded in the PETSc library. If you'd rather look at the traditional profiling/log data dumped by PETSc, you can use the ``-log_view`` flag to have PETSc print that information to the standard output. Interpreting Profile Data with TDyProf ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you've generated a ``tdycore_profile.csv`` file, you can use a tool called ``tdyprof`` (located in the ``tools/`` subdirectory of the source tree). This Python script digests the contents of the CSV file you give it and generates nicely-formatted reports for desired information. Use it thus::: tdyprof [options] or just type ``tdyprof`` by itself to see its usage information. For example, to see the top 10 "hotspots" in the performance profile::: tdyprof tdycore_profile.csv top10 tdyprof: showing top 10 hits: Stage Name Event Name Time FLOP Main Stage TDyTimeIntegratorRunToTime 0.139049 1.85991e+07 Main Stage SNESSolve 0.13895 1.85991e+07 Main Stage SNESJacobianEval 0.120045 609812. Main Stage TDyMPFAOSNESJacobian_3DMesh 0.120032 609812. Main Stage TDyMPFAOIJacobian_Vertices_3DMesh 0.118853 606312. TDycore Setup TDyDriverInitializeTDy 0.0912797 52040. TDycore Setup TDySetup 0.0533342 52040. TDycore Setup TDyMPFAOInitialize 0.0533217 52040. Main Stage DMPlexDistribute 0.0386017 0. TDycore Setup TDyCreateJacobian 0.0370897 0. Generating Scaling Study Plots with TDyPerfPlot ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TODO