Sending and Receiving Messages

• The basic functions for sending and receiving messages in MPI are the MPI_Send and MPI_Recv, respectively.
• The calling sequences of these routines are as follows:

  int MPI_Send(void *buf, int count, 
               MPI_Datatype datatype,
               int dest, int tag, 
               MPI_Comm comm)
  int MPI_Recv(void *buf, int count, 
               MPI_Datatype datatype,
               int source, int tag, 
               MPI_Comm comm, 
               MPI_Status *status)
  

• MPI_Send sends the data stored in the buffer pointed by buf.
• This buffer consists of consecutive entries of the type specified by the parameter datatype.
• The number of entries in the buffer is given by the parameter count.

Table 2: Correspondence between the datatypes supported by MPI and those supported by C.

MPI Datatype	C Datatype
MPI_CHAR	signed char
MPI_SHORT	signed short int
MPI_INT	signed int
MPI_LONG	signed long int
MPI_UNSIGNED_CHAR	unsigned char
MPI_UNSIGNED_SHORT	unsigned short int
MPI_UNSIGNED	unsigned int
MPI_UNSIGNED_LONG	unsigned long int
MPI_FLOAT	float
MPI_DOUBLE	double
MPI_LONG_DOUBLE	long double
MPI_BYTE
MPI_PACKED

Note that for all C datatypes, an equivalent MPI datatype is provided.

• MPI allows two additional datatypes that are not part of the C language.
• These are MPI_BYTE and MPI_PACKED.
  • MPI_BYTE corresponds to a byte (8 bits)
  • MPI_PACKED corresponds to a collection of data items that has been created by packing non-contiguous data.
• Note that the length of the message in MPI_Send, as well as in other MPI routines, is specified in terms of the number of entries being sent and not in terms of the number of bytes.
• Specifying the length in terms of the number of entries has the advantage of making the MPI code portable,
• since the number of bytes used to store various datatypes can be different for different architectures.

• The destination of the message sent by MPI_Send is uniquely specified by
  • dest argument. This argument is the rank of the destination process in the communication domain specified by the communicator comm.
  • comm argument.
• Each message has an integer-valued tag associated with it.
• This is used to distinguish different types of messages.
• The message-tag can take values ranging from zero up to the MPI defined constant MPI_TAG_UB (implementation specific, at least 32767).

• MPI_Recv receives a message sent by a process whose rank is given by the source in the communication domain specified by the comm argument.
• The tag of the sent message must be that specified by the tag argument.
• If there are many messages with identical tag from the same process, then any one of these messages is received.
• MPI allows specification of wild card arguments for both source and tag.
  • If source is set to MPI_ANY_SOURCE, then any process of the communication domain can be the source of the message.
  • Similarly, if tag is set to MPI_ANY_TAG, then messages with any tag are accepted.

• The received message is stored in continuous locations in the buffer pointed to by buf.
• The count and datatype arguments of MPI_Recv are used to specify the length of the supplied buffer.
• The received message should be of length equal to or less than this length.
• This allows the receiving process to not know the exact size of the message being sent.
• If the received message is larger than the supplied buffer, then an overflow error will occur, and the routine will return the error
  MPI_ERR_TRUNCATE.

• After a message has been received, the status variable can be used to get information about the MPI_Recv operation.
• In C, status is stored using the MPI_Status data-structure.
• This is implemented as a structure with three fields, as follows:

  typedef struct MPI_Status { 
    int MPI_SOURCE; 
    int MPI_TAG; 
    int MPI_ERROR; 
  };
  

• MPI_SOURCE and MPI_TAG store the source and the tag of the received message.
• They are particularly useful when MPI_ANY_SOURCE and MPI_ANY_TAG are used for the source and tag arguments.
• MPI_ERROR stores the error-code of the received message.

• The status argument also returns information about the length of the received message.
• This information is not directly accessible from the status variable, but it can be retrieved by calling the MPI_Get_count function.
• The calling sequence:

  int MPI_Get_count(MPI_Status *status, 
                    MPI_Datatype datatype, 
                    int *count)
  

• MPI_Get_count takes as arguments the status returned by MPI_Recv and the type of the received data in datatype, and returns the number of entries that were actually received in the count variable.

 $$

The MPI_Recv returns only after the requested message has been received and copied into the buffer.

 $$

That is, MPI_Recv is a blocking receive operation.

 $$

However, MPI allows two different implementations for MPI_Send.

1

MPI_Send returns only after the corresponding MPI_Recv have been issued and the message has been sent to the receiver.

2

MPI_Send first copies the message into a buffer and then returns, without waiting for the corresponding MPI_Recv to be executed.

 $$

In either implementation, the buffer that is pointed by the buf argument of MPI_Send can be safely reused and overwritten.

• MPI programs must be able to run correctly regardless of which of the two methods is used for implementing MPI_Send.
• Such programs are called safe.
• In writing safe MPI programs, sometimes it is helpful to forget about the alternate implementation of MPI_Send and just think of it as being a blocking send operation.