Behavioral studies indicate that directional gaze and hand pointing are fundamental social signals that may capture spatial attention more powerfully than directional arrows. By using fMRI, we explored whether reflexive shifts of attention triggered by different distracters were influenced by the motor effector used for performing an overt response. In separate blocks, healthy participants performed a directional saccadic or a hand pointing movement. Color changes of a central black fixation point constituted the imperative instruction signal to make a leftward (red color) or a rightward (blue color) movement while ignoring distracting leftward or rightward oriented gaze, hand pointing, or arrow. Distracters that were directionally incongruent with the instruction cue impaired the saccadic and pointing-release RTs. The comparison of incongruent vs. congruent conditions showed an increase of BOLD signal in the frontal eye field (FEF), the intraparietal sulcus (IPS), and the posterior parietal cortex (PPC) bilaterally. Importantly, a specific relationship between distracter and effector used for the response was found in these frontal and parietal regions. In particular, higher activity in the FEF, for distracting gaze was found mainly during the saccadic response task. In the same vein, higher activity in the left and right IPS regions was found for the distracting hand mainly in the hand pointing task. The results suggest that reflexive shifts of attention triggered by social signals are coded in the fronto-parietal cortex according to effector-specific mapping rules.