Solved.

Thanks!

One way to do it:

If the user input is a quaternion, convert it to axis and angle.

Convert axis and angle to a rotation vector: vec3 rv = axis * angle

Project this vector to your constraint axis unit vector: vec3 constrained_rv = constraintAxis * constraintAxis.Dot(rv);

Then convert result back to axis / angle: float angle = constrained_rv.Length(); vec3 axis = constrained_rv / angle;

Then convert to quat or matrix or whatever you need: quat q; q.FromaxisAndAngle(axis, angle);

So that's quite simple. I assume everything is given in global space, otherwise perform the operations in the local space of the joint for example.

Ask if there's something unclear...

Thanks for your reply!

22 minutes ago, knsl96 said:

float dprod = constraintAxis.Dot(constraintAxis, rv);

This line already gives the angle, so no need to calculate magnitude later. But that's just an optimization.

Pretty sure the NaN happens here if angle is zero:

24 minutes ago, knsl96 said:

MyVector axis = new MyVector(constrained_rv._x / angle, constrained_rv._y / angle, constrained_rv._z / angle); 

So you would need to return identity quaternion if angle is close to zero before that.