I want to make the vehicle steer to look at the player, so if the player is on the vehicle right side, I should call "vehicle->SteerRight()", if the player is on the left I should call "vehicle->SteerLeft()"
Given (D3DXVECTOR3 PlayerPosition and D3DXMATRIX VehicleWorldMatrix) how do I find out if the vehicle should steer right or left?
If PlayerPosition is in world space then you just need to do inverse transform of it's position by vehicle matrix. After that sign of Y vector will tell you if he is on the left or right side.
Another way of doing this, that I personally find more intuitive, is to put a plane that runs through the vehicle and essentially cuts it in a left and a right half. Then you can just detect which side of the plane the player is on, which is usually done through a signed distance check against the plane. Positive might mean left and negative right, depending on which side of the plane you consider "up".
@BoredEngineer: Can you please provide code example?
For directx you need to apply this method:
on vehicle world matrix and then transform position of the player by inverted matrix.
If you have a forward vector of the vehicle, you could also project playerpos, vehiclepos and forward_vector to the 2D ground plane and do:
(playerpos - vehiclepos) dot forward_vector
and check the sign.
Projecting to 2D ground plane just means removing the y coordinate from all coordinates. (or z, or whatever is "up" in your world)
Might not work like you want if you want to be driving on walls and ceilings, but if you don't need that, it should work.
I'm not sure if this is correct, but here is what I'm doing now and it seems that it works:
D3DXVECTOR3 view( -VehicleWorldMatrix.m[2][0], -VehicleWorldMatrix.m[2][1], -VehicleWorldMatrix.m[2][2] );
D3DXVec3Normalize(&view, &view);
D3DXVECTOR3 toTarget = PlayerPosition - VehiclePosition;
D3DXVECTOR3 cross;
D3DXVec3Cross(&cross, &toTarget, &view);
if (cross.y > 0)
{
// Steer right
} else {
// Steer left
}
view would be what I called "forward_vector".
Your method should work fine in most situations, with the same limitations as what I posted.
just a bit more unnecessary calculations compared to my suggestion 
BoredEngineers solution is the most general purpose and would work regardless of the orientation of the vehicle. (assuming you want "right of" to mean from the vehicles perspective, which from an outsides observer would be "above" or "below" the vehicle, if it was turned on its side)
In your and my solution, the roll of the vehicle (or rotation around the length of the vehicle, or whatever you want to call it) will not affect what is "right of" and "left of" the vehicle.
No need to normalize the "view" vector btw, since you are only interested in the sign of y.
@Olof Hedman:
BoredEngineers solution is the most general purpose and would work regardless of the orientation of the vehicle. (assuming you want "right of" to mean from the vehicles perspective, which from an outsides observer would be "above" or "below" the vehicle, if it was turned on its side)
D3DXMatrixInverse is taking FLOAT* pDeterminant,
I have: D3DXVECTOR3 PlayerPosition; and D3DXMATRIX VehicleWorldMatrix
How do I call D3DXMatrixInverse() ?
Read the docs. "Pointer to a FLOAT value containing the determinant of the matrix. If the determinant is not needed, set this parameter to NULL."
