What am I doing wrong when trying to calculate a ray from the camera?

glm::vec4 mouseClip = {ev.cursor.x * 2 / 640.f - 1, 1 - ev.cursor.y * 2 / 480.f, 0, 1};

glm::mat4 inv = glm::inverse(game::camera->projection * game::camera->transform);

glm::vec4 p = inv * mouseClip;
p.x /= p.w;
p.y /= p.w;
p.z /= p.w;

glm::vec3 direction = glm::normalize(glm::vec3{p});

I can guarantee that mouseClip's x and y is correct (from [-1, -1] to [1, 1]).

glm::vec3 intPos, intNor;
if(glm::intersectRaySphere(glm::vec3{game::camera->transform[3]}, direction, glm::vec3{0, 0, 0}, 1, intPos, intNor)) {
    printf("intersect\n");
}

intersectRaySphere does as it says, here is it's documentation.

Problem is that.. well.. basically everything is off. There is nothing I can accurately describe, it seems as if the camera's transformation is always exaggerated. I know this is all very vague, please ask questions if you have any.

I tried using glm::unProject instead, but issues still exist:

glm::vec3 direction = glm::normalize(glm::unProject(glm::vec3{ev.cursor.x * 2 / 640.f - 1, 1 - ev.cursor.y * 2 / 480.f, 1}, game::camera->transform, game::camera->projection, glm::vec4{0, 0, 640, 480}));

As I said, I can guarantee the mouse position is correct, that's not the issue.

I apparently have passed a few wrong arguments to glm::unProject, here is the updated code:

glm::vec3 direction = glm::unProject(glm::vec3{ev.cursor.x, 480 - ev.cursor.y, 1}, glm::mat4{1}, game::camera->projection * game::camera->transform, glm::vec4{0, 480, 640, -480});
direction = glm::normalize(direction - glm::vec3{game::camera->transform[3]});

And now it works for almost all cases. It starts to break if there's any Y translation, or any rotation at all, or if I perform glm::translate and glm::rotate in the wrong order. Those values seem to be always exaggerated too much for seemingly no reason.

The Y translation making it incorrect? That was a non-issue, I was inverting the mouse's Y coordinate twice, durr.
But rotation in the view matrix is still making it incorrect, said rotation (on any axis) also seems to be exaggerated, as if it's scaled up.

Fixed the issue. And, as ALWAYS it's the stupidest issues you can't even imagine.

Check the unProject documentation:

Quote

detail::tvec3<T> glm::gtc::matrix_transform::unProject(detail::tvec3< T > const& win,
		detail::tmat4x4< T > const& model,
		detail::tmat4x4< T > const& proj,
		detail::tvec4< U > const& viewport 
)

 

The second argument? That's the modelview matrix, not just the view matrix! So I just ended up passing the inverse of the view matrix there, and it works fantastically now.

It's GLM's documentation that's misleading. The program now works without issues.

It seems to be correct in 0.9.9:

unproject