4 Example

The following is a somewhat simplified version of the code I use in my simple NBODY1 like code with multiprocessor.

 
.....
    g6_setmccount(nprocs);
    g6_open(grape6_id);
.....

void particle_system::update_grape_data(int  n_next)
{
    vector j218 = vector(0.0,0.0,0.0);
    g6_change_cbjpmode(grape6_id, 1);
    MPI_Barrier(MPI_COMM_WORLD);
    g6_reset(grape6_id);
    g6_set_ijp_mode(grape6_id, 1);
    for (int i = 0; i< 200; i++) g6_dummywait();
    g6_reset(grape6_id);
    for (int i = 0; i < n_next; i++) {
        particle *bi = nt[i].pptr;
        vector j6 = ONE_SIXTH*bi->get_jerk();
        vector a2 = 0.5L*bi->get_acc();
        g6_set_j_particle(grape6_id,
                           bi->get_grape_index(),
                           bi->get_index(),
                           bi->get_time(),
                           bi->get_timestep(),
                           bi->get_grape_mass(),
                           (real*)&j218,
                           (real*)&j6,
                           (real*)&a2,
                           (real*)bi->pget_vel(),
                           (real*)bi->pget_pos());
    }

    g6_set_ijp_mode(grape6_id,0);
    MPI_Barrier(MPI_COMM_WORLD);
    g6_change_cbjpmode(grape6_id,0);
    MPI_Barrier(MPI_COMM_WORLD);
    g6_reset(grape6_id);

}

The first function, g6_setmccount tells the GRAPE-6 library what configuration it should use when asked to send $j$-particles. We can call it with either nprocs equal to 1, 2, or 4. This argument of course must be the same as the number of processors we actually use. In the case of MPI, most likely one would use MPI_Comm_size to get the number of processors. This function can be called anytime, even before calling g6_open.

The next function, update_grape_data, shows the full code to update $j$-particles after one blockstep in the case of individual timestep (blockstep) integrator. It first switches from N-mode to M-mode, and then actually sends $j$-particles and finally switches back to N-mode.

This part:

    g6_change_cbjpmode(grape6_id, 1);
    MPI_Barrier(MPI_COMM_WORLD);
    g6_reset(grape6_id);
    g6_set_ijp_mode(grape6_id, 1);
    for (int i = 0; i< 200; i++) g6_dummywait();
    g6_reset(grape6_id);

is the code to switch to M-mode. The first function, g6_change_cbjpmode(grape6_id, 1), actually changes the mode. If the second argument is one, it lets the NB to M-mode, and if 0, to N-mode.

The remaining functions essentially just reset the hardware after the mode is changed. Note that we are now in message-passing environment. The host computers need to perform barrier synchronization after g6_change_cbjpmode(grape6_id, 1) is called, because otherwise other hosts might not have finished the call.

The call to g6_set_ijp_mode(grape6_id, 1) change the internal mode of each GRAPE-6 board (now in M-mode, which means one host is now connected to GRAPE-6 boards under different two or four different NBs) to the state to receive $j$-particle data. I believe this function is not really necessary, since it is called from g6_set_j_particle anyway, but I have not yet tested the code without this function called here...

The next line, the call to g6_reset(grape6_id), is to reset the hardware after mode change. This call is necessary since changing the mode can put the system into a strange state.

The call to g6_set_ijp_mode(grape6_id, 1) takes effect in something like 1 microsecond after executed. The next dummywait loop is to spin-wait about that amount of time. This is of course a rather dangerous way, since whether this works or not depends on the speed of the host computer. When the host computer is replaced, this code might suddenly stop working. The final call to g6_reset is also probably not necessary.

The second loop simply send $j$-particles, in the way exactly the same as in the case of single-host code.

The final part switches back to M-mode.

 
    g6_set_ijp_mode(grape6_id,0);
    MPI_Barrier(MPI_COMM_WORLD);
    g6_change_cbjpmode(grape6_id,0);
    MPI_Barrier(MPI_COMM_WORLD);
    g6_reset(grape6_id);

Here, again, the call to g6_set_ijp_mode(grape6_id,0) is probably unnecessary. Before we call g6_change_cbjpmode(grape6_id,0), we need to perform barrier synchronization because even when one host has finished sending its $j$-particle data, others might still be sending data, and if so, it should wait all other hosts to finish sending data before switching back to M-mode. Also, after calling g6_change_cbjpmode, another barrier synchronization is necessary. If we omit this synchronization one host might send some data before some other host changes its NB from M-mode to N-mode. Thus, even if one host has its mode changed, the data it sends might still be received by GRAPE-6 boards under some other hosts, resulting in some unpredictable effects.