There are several quite important guidelines which must be followed to obtain meaningful result in parallel host applications. I'll describe what I'm aware of in the following.
The number of 
particles on all host must exactly be the same. For
example, when the total number of particles is 1024 and you use 4
hosts, the number of 
-particles to be taken care by each host must
be 256. If the number of particles is not an integer multiple of the
number of hosts, you have to add dummy particles (with zero mass is
fine), so that all hosts have the same number of particles.
This is simply because the argument nj for the firsthalf/lasthalf pair is the same for all boards, while they receive the data from different hosts.
The GRAPE-6 hardware uses the particle index to avoid self-interaction
and also for neighbor calculations. Therefore, the index must be unique
over the whole 
-body system. One has to assign index so that any of
two particles, whether or not they are on the same host or not, must
have different indices.
When a host recognizes a hardware error returned from GRAPE-6, the error may be due to the data sent from some other host. Therefore, the simplest way of the recovery from the error is to let ALL hosts perform the recovery procedure when ANY of the hosts detects error.
If more than two multi-host programs are running simultaneously, it is possible that they go into the deadlock state. For example, if program A and B both use g6host0 and g6host1, they need to acquire GRAPE-6 on both machine. Thus, if, for example, on g6host0 A acquires GRAPE-6 and on g6host1 B acquires GRAPE-6, A will be waiting B to release GRAPE-6 on g6host1 and B will be waiting A to release GRAPE-6 on g6host0. Either program cannot go further.
Therefore, some mechanism which prevents such deadlock from taking place is absolutely necessary. Such a mechanism is actually quite simple to implement. If both of programs A and B first try to acquite GRAPE-6 on g6host0, and proceed to g6host1 only if it acquired GRAPE-6 on g6host0, there will be no deadlock. Therefore, a code like the following (Note that this is UNTESTED yet!) should do the work.
int i, numprocs, myid;
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
MPI_Barrier(MPI_COMM_WORLD);
for(i=0;i<numprocs; i++){
if (myid == i) g6_open(0);
MPI_Barrier(MPI_COMM_WORLD);
}