The current implemntation of nbody relies on the Morton key encoded into a single long int word. Actual number of bits used (per dimension) is defined in the file BHtree.h as integer constant default_key_length. Set this value not exceeding 10 on machines long is of 32-bit length. On machines with 64-bit long, you can set its value to up to 21.
The value of default_key_length determines the maximimum
possible depth of the tree. Unless you are using really very large
number of particles (say, 
or more), the maximum of 10 levels is
probably enough.
If the maximum depth is not suffucient to resolve the distribution of particles, nbody just stops subdividing the cell there. As a result, the lowest cell may contain rather large number of particles. This does not cause any integration error, but causes some performance degradation.
Currently, nbody stops if the memory allocated for tree turned out to be insufficient to build the tree. One could change the program so that either it reallocates a larger amount memory or stop to refining the tree at that point. It's not clear which approach is better, and neither option is implemented yet.