Library Implementation

Here we briefly describe the structure and build process of the library prototype, the common utilities, along with existing and proposed language support.

Structure and Build Process

Below figure illustrates the directory structure of the Mamba library. At the top level, build files and a README file are provided, whilst most of the Mamba implementation is in the common and memory directories. Language specific files for Fortran and C++ are stored in the fortran and cpp directories. Dependent libraries for loop analysis, stored in the loopanalyzer subdirectory, are imported via git submodules via autogen.sh if required.

An illustration of the directory structure of the Mamba library.

An illustration of the directory structure of the Mamba library.

The Mamba library exploits the autotools build system for compilation; each of the code subdirectories is built as a convenience library, and these are combined to form libmamba.

mkdir build;
cd build;
// Only required if you want to use the --with-loop-analysis configure arg
// If not, you can use ‘autoreconf -i‘ instead
../autogen.sh
../configure [--prefix=/p/a/t/h]    \
             [--with-sicm=/p/a/t/h] \
             [--with-umpire=/p/a/t/h] \
             [--with-jemalloc=/p/a/t/h] \
             [--with-jemalloc-prefix=<prefix>] \
             [--with-memkind=/p/a/t/h] \
             [--with-fortran] \
             [--with-cpp] \
             [--with-loop-analysis] \
             [--enable-embedded] \
             [--enable-cuda[=yes|no|<arch>]] \
             [--enable-hip-rocm[=yes|no]] \
             [--enable-opencl[=yes|no]] \
             [--with-opencl=/p/a/t/h] \
             [--enable-discovery[=yes|no|default]];

make;
make check-tests;
make check-examples;
make install; (optional)

Common Utilities

Library Initialisation

The Mamba library must be initialised before first use, and de-initialised after final use. To this end, the library includes functions for initialisation, finalisation, and checking the state of the library, in a similar way to the MPI library.

mmbError mmb_init(mmbOptions *in opts);
mmbError mmb_is_initialized(int *initialized);
const char * mmb_version();
mmbError mmb_finalize();

The mmbOptions structure allows the user to provide initialisation options for the Mamba library, and may be constructed and modified as shown in the code snippet below. The user may also provide MMB_INIT_DEFAULT as the argument to mmb_init for default initialisation.

mmbError mmb_options_create_default(mmbOptions **opts);
mmbError mmb_options_destroy(mmbOptions *opts);
mmbError mmb_options_set_debug_level(mmbOptions *opts, int_level);
mmbError mmb_options_set_user_log_func(mmbOptions *opts, mmbUserLogFunc user_log_func);
// ...

Convenience Data Structures

Mamba includes various convenience data structures for interacting with the library. Commonly used in the public C API are two wrapper structures for a simple resizeable array used to store either a list of dimensions, or a list of indices, * mmbDimensions * mmbIndex

An excerpt of the API available for this type of data structure is as follows:

mmbError mmb_dimensions_create(const size_t ndim, mmbDimensions **out_dims);
mmbError mmb_dimensions_set(mmbDimensions *in_dims, size_t dimN, ...);
mmbError mmb_dimensions_create_fill(const size_t ndim, const size_t* values, mmbDimensions **out_dims);
mmbError mmb_dimensions_copy(mmbDimensions *dst, const mmbDimensions *src);
mmbError mmb_dimensions_copy_buffer(size t *dst, const mmbDimensions *src);
mmbError mmb_dimensions_fill(mmbDimensions *dst, const size_t *src, const size_t nelt);
mmbError mmb_dimensions_resize(mmbDimensions *in_dims, const size_t ndim);
mmbError mmb_dimensions_destroy(mmbDimensions *in_dims);
int mmb_dimensions_cmp(const mmbDimensions *d1, const mmbDimensions *d2);
mmbError mmb_dimensions_prod(const mmbDimensions *in dims, size_t *out_prod);
mmbError mmb_dimensions_get_size(const mmbDimensions *in_dims, size_t *size);
mmbError mmb_dimensions_get_sizeof(const mmbDimensions *in_dims, size_t *size);

Logging

Mamba includes a logging infrastructure (./common/mmb logging.h/c), used internally and available for users to exploit. Maximum logging verbosity is defined by default with a compile time switch, and reduced verbosity may be requested by setting the environment variable MMB_LOG_LEVEL to one of the following values:

/** log level for errors */
#define MMB_LOG_ERR 0
/** log level for warnings */
#define MMB_LOG_WARN 1
/** log level for informational messages */
#define MMB_LOG_INFO 2
/** log level for debugging messages */
#define MMB_LOG_DEBUG 3
/** log level for really chatty logging */
#define MMB_LOG_NOISE 4

/** Usage in Mamba, also available for use in user codes */
/** Error messages */
MMB_ERR(format, ...)
/** warning messages */
MMB_WARN(format, ...)
/** informational messages */
MMB_INFO(format, ...)
/** debug messages */
MMB_DEBUG(format, ...)
/** chatty messages */
MMB_NOISE(format, ...)

/** example usage */
ERR("Error with integer value: %d\n", errno)
/** example output in format:
    [error type] (process id): function(filename:line) error message */
[E] (12083): main(example.c:25) Error with integer value: 1

The Mamba logging infrastructure by default outputs to stderr, however may also be initialised with a user-provided logging function to override this behaviour, as seen in e.g. the user defined logging example code.

Error Handling

Error handling is managed via datatype mmbError, used as the return type for almost all Mamba API functions, and demonstrated in the code snippet below. As demonstrated, mmb_error description may be used to acquire a const char* string description of the error, which should not be free’d by the user.

mmbError mmb_example_function_wrapper(void) {

   mmbError err = mmb_example_function(arg1, arg2);
   if(err != MMB OK) {
       MMB ERR("Failed to run the example function: %s\n", mmb_error_description(err));
       goto BAILOUT;
   }
   /** ... */

BAILOUT:
   return err;
}

Language Support

The majority of the library is written in C, based on the C11 language standard. As evidenced throughout the code listings in this document, we adhere to a programming style where:

  • Public API functions are prefixed mmb_

  • Functions return an mmbError type for error handling.

  • Function names are written in underscore case.

  • Typenames are written in camel case.

  • Constructors are formatted mmb_(object_name)_create, with an optional further configuration postfix (e.g. _2d) for overloading constructors.

  • Destructors are formatted mmb_(object_name)_destroy

  • Object memory allocation/free routines are similarly formatted mmb_(object_name)_lloc, mmb_(object_name)_free.

  • Commenting in doxygen format is encouraged

In this section we detail the Fortran and C++ specific interfaces to Mamba.

Fortran

The Fortran interface is designed to mirror that of the C API and in most cases uses identically-named types to the C API for declaration of opaque objects that are passed to Mamba API procedures. The Fortran interface makes use of the interoperability features introduced in the Fortran 2008 revision of the Fortran standard. In line with the expecation for Fortran the Fortran API uses 1-based indexing for array tile and tile index spaces. The Fortran interface requires the use of the mamba module to provide the definitions of the API subprograms and support types. The module is used as follows:

use mamba

In addition to required types, a set of kind values are defined for integer declarations: mmbErrorKind for the return error value, mmbSizeKind for memory counts and mmbIndexKind for quantities that relate to array indices.

The Fortran interface uses subroutines instead of functions with the last argument used to return an error code, this code will either be MMB_OK, MMB_ERROR or one of the many specific error values. Returning the error code is optional as shown in the finalize call below.

use mamba
integer(mmbErrorKind) err
call mmb_init(err)
call mmb_finalize()

Some subroutines accept a missing optional argument in the case that the equivalent C API function would accept a NULL. This may mean that keyword arguments may be needed to disambiguate arguments. For example in the below code fragment the error argument is optional in the first call but the chunks argument is optional in the second call (in which case the whole array is tiled).

integer(mmbErrorKind) :: err
call mmb_array_tile(mamba_array_a, chunks)
call mmb_array_tile(mamba_array_b, err=err)

In cases where the Mamba C API provides a mechanism to allocate and deallocate objects which are passed around the API but whose components are not accessed directly the Fortran inteface uses an opaque type. The opaque types are mmbMemSpace, mmbMemInterface, mmbLayout, mmbLayoutPadding, mmbArray, mmbTiling, mmbTileIterator and mmbIndex. Variables using these types can be defined as shown in the example code below:

use mamba
type(mmbArray) :: mamba_array
type(mmbLayout) layout

New derived types are provided for objects where direct access to the components is needed, this is true for example for the mmbArrayTile type which provides (in part) corresponding elements to the mmbArrayTile structure in the C API. The mmbArrayTile type is defined as follows:

TYPE :: mmbArrayTile
   TYPE(C_mmbArrayTile) :: c tile ! The C tile structure
   INTEGER :: rank=0
   TYPE(C_PTR) :: ptr
   INTEGER(mmbIndexKind), allocatable :: lower(:)
   INTEGER(mmbIndexKind), allocatable :: upper(:)
   INTEGER(mmbIndexKind), allocatable :: alower(:)
   INTEGER(mmbIndexKind), allocatable :: aupper(:)
   INTEGER(mmbIndexKind), allocatable :: dim(:)
   INTEGER(mmbIndexKind), allocatable :: abs dim(:)
   LOGICAL is_contiguous
END TYPE mmbArrayTile

The components lower, upper, alower, aupper, dim and abs_dim are directly equivalent to those of the ArrayTile object in the C API but the lower and upper boulds are 1-based and not 0-based. The rank component holds the rank (number of dimensions) of the tile and the is_contiguous component is true if the tile is contiguous in memory. The other components are used internally in the API.

Various structures and preprocessor macros are used in the C API to pass configuration options into the API functions, notably for memory registration and requests for spaces and interfaces. The Fortran API defines types with the same names, for example for mmbMemSpaceConfig, mmbSizeConfig, mmbMemInterfaceConfig, mmbProviderOptions, MMB_MEMINTERFACE_CONFIG_DEFAULT, and MMB_MEMSPACE_CONFIG_DEFAULT``. These types can be used to construct the relevant constants for initialization.

use mamba
integer(mmbErrorKind) err
type(mmbMemSpace) dram_space
type(mmbMemSpaceConfig) dram_config
type(mmbMemInterface) dram_interface
type(mmbArray) :: mba0,mba1

call mmb_init(err=err)
dram_config = mmbMemSpaceConfig(mmbSizeConfig(MMB_SIZE_SET,.false.,8000),&
                             MMB_MEMINTERFACE_CONFIG_DEFAULT)
call mmb_request_space(MMB_DRAM, MMB_EXECUTION_CONTEXT_DEFAULT, &
                     new_space=dram_space, err=err )
call mmb_request_interface(dram_space, new_interface=dram_interface, err=err)

The mmbSizeConfig arguments relate to the action, a logical to indicate if the is a numa configuration and the size requested for the space.

An mmbDimensions object is not used in the Fortran API as a Fortran array can be used directly. All API functions that accept a Dimensions object will accept an array instead.

use mamba
type(mmbArray) :: mamba_array
integer(mmbIndexKind), dimension(2) :: dims,chunks
integer(mmbSizeKind), allocatable, dimension(:) :: tiling_dims
! ...
dims = [array_size_M, array_size_N]
call mmb_array_tile(mamba_array, dims, err)
! enquire about tiling dimentions
allocate( tiling_dims(2) )
call mmb_tiling_dimensions(mamba_array, tiling_dims, err)

The iterator routines mmb_tile_iterator_first and mmb_tile_iterator_next have an additional argument that returns the relevant tile of type mmbArrayTile. (The C API allowed direct access to the tile from the Iterator). The tile accessor routines mmb_tile_at, mmb_tile_at_2d etc. also return an mmbArrayTile object.

In contrast to the C API the Fortran interface associates a Fortran pointer with the tile data using the subroutine mmb_tile_get_pointer. The following code snippet illustrates obtaining the pointer from the tile and then using it to set the data covered by the tile:

real, pointer, dimension(:,:) :: tp
call mmb_tile_get_pointer(tile, tp)
tp(tile%lower(1):tile%upper(1),tile%lower(2):tile%upper(2))=1.0

Note that the returned pointer may actually point to a larger space than that which should be accessed via the lower/upper bounds.

The Fortran Interface also supports access to tile elements via preprocessor macros. This is not a standard paradigm for Fortran but it is implemented in case there are distributions that cannot be dealt with using a Fortran pointer. To use the index macros the program must include the file mambaf.h in addition to using the mamba module. This makes the indexing macros available and they can be used as:

use mamba
#include "mambaf.h"

! tiles tile_a, tile_b, tile_c defined above
do ei = tile_c%lower(1),tile_c%upper(1)
 do ej = tile_c%lower(2),tile_c%upper(2)
  do ek = tile_a%lower(2),tile_a%upper(2)
    MMB_IDX_2D(tile_c,tp_c,ei,ej) = &
    MMB_IDX_2D(tile_c,tp_c,ei,ej) + &
    MMB_IDX_2D(tile_a,tp_a,ei,ek) * MMB_IDX_2D(tile_b,tp_b,ek,ej)
  end do
 end do
end do

Note that the macro accepts as arguments the array tile, the tile pointer and then the list of indexes. Additional macros MMB_IDX_1D and MMB_IDX_2D_NORM having the same meaning as per the C interface.

The following code shows part of one of the Mamba Fortran example programs with comments removed. The example allocates a buffer, registers it with mamba and then creates a layout and mamba array.

program array_copy_wrapped_1d
  use mamba
  implicit none
  integer, parameter :: M=128
  integer(mmbErrorKind) err
  integer(mmbSizeKind) ntiles
  integer(mmbSizeKind), allocatable, dimension(:) :: tile_dims
  type(mmbMemSpace) dram_space
  type(mmbMemInterface) dram_interface
  type(mmbArray) :: mba0
  integer(mmbIndexKind) :: dims(1)
  type(mmbLayout) layout
  type(mmbTileIterator) it
  type(mmbArrayTile) tile
  real, dimension(:), allocatable,target :: buffer0
  real, dimension(:), pointer :: tp
  integer :: i,itile,chunksize

  allocate( buffer0(m) )

  call mmb_init(err)
  dram_config = mmbMemSpaceConfig(mmbSizeConfig(MMB_SIZE_SET,.false.,8000),&
                            MMB_MEMINTERFACE_CONFIG_DEFAULT)
  call mmb_register_memory(MMB_DRAM, MMB_EXECUTION_CONTEXT_DEFAULT, &
                         dram_config,err=err)
  call mmb_request_space(MMB_DRAM, MMB_EXECUTION_CONTEXT_DEFAULT, &
                       new_space=dram_space, err=err )
  call mmb_request_interface(dram_space, new_interface=dram_interface, err=err)
  call mmb_layout_create_regular_1d(int(storage_size(buffer0)/8,mmbSizeKind),&
                     mmb_layout_padding_create_zero(), &
                     layout,err)
  call mmb_array_create_wrapped(buffer0,dims, layout, &
                               dram_interface, MMB_READ_WRITE, mba0,err )
  call mmb_array_tile(mba0, dims, err)
  call mmb_tile_iterator_create(mba0, it, err)
  call mmb_tile_iterator_first(it,tile,err)

  block
     asynchronous :: buffer0
   do i=1,M
     tp(i)=i*0.001+3.0
   end do
  end block

  call mmb_finalize(err)

end program array_copy_wrapped_1d

A mamba iterator is used to get the first tile in the array, its location in the larger array can be determined from elements of the mmbTile type. This type also provides a pointer to the data (in this example we illustrate the initial prototype where this is the full array).

Fortran examples are provided with the Mamba distribution and are a useful resource that illustrates the features of the API in a practical context.

C++ Interface

The current C++ interface is a C++ wrapper of the C interface, due to the similarity of the languages this is relatively simple to provide, and as such the C interface design is sufficient to describe current interactions in C++.

Work is on-going to extend this interface to use advanced features of C++ to provide a cleaner interface more suited to C++ programming. This includes:

  • Replacing function variants with overloaded functions (e.g. _nd function suffixes).

  • Wrapping the C Mamba array in a C++ Mamba array template class, allowing users to specify the underlying type of the data on construction.

  • Wrapping the C Mamba array tile object in a C++ Mamba array tile template class, allowing users to avoid specifying type during tile access.

  • General usage of C++ syntactic sugar to ensure the Mamba API is natural to use for C++ programmers.

In the final stages of the project, we will also explore where possible other potential improvements to Mamba, such as understanding the effect of overloaded array operators for tile indexing, while ensuring to maintain the possibility of compiler vectorization of the array accesses, and investigate the applicability of more modern C++ language features, for example potentially storing loop kernels as lambdas in tile iterators. This is not a major focus of the project, as many such features are being explored in C++ centric libraries such as Kokkos.

Examples and Tests

A set of examples are included with the library, demonstrating various features of the Mamba library for users. They are stored in the examples/ subfolder, separated by language, and automatically built and installed during make install. Each example is documented in the library README, and can either be run individually from the installed directory, or all examples may be run sequentially at build-time as a secondary set of tests via make check examples.

A set of unit tests are included in the tests/ subfolder, targeting the C-based library implementation code. These may be run at build time via make check.