I would be thinking in terms of just calculate the position of the arrow heads of the gizmo in world space, or screen space, and just pick the mouse coords as a ray in world space or a point in screen space, and pick the nearest gizmo thingy / bounding box?

You can use the same matrices you use for rendering to get them into world space / screen space (same as software transforming a vertex).

I'm not sure if I got your question right. But I'll write it anyway, maybe some part of it would be helpful.

I don't think for it to work in every viewport is not even relevant here. Basically, what you have here is an MVP matrix, that provides you with space transformation for different camera positions and perspectives. The problem that you have is that you want to be able to take some MVP transformed primitive, find a point or a sector on it transform back into the model space and make something with this data, highlight an area or whatever. That's how I got that.

What I would do, I would cast a ray in the MVP space, the ray origin would be on the on the nearest frustum plane with respect to mouse coordinates, that ray would be normal to that plane. Then I would get a point of contact, if the ray hits the primitive, in the MVP space, and from this point, I could just take inverse MVP matrix and transform this point back to the model space. Then I would have my primitive and a point of contact that would be on the surface of this primitive. And now one I could do whatever I want with it, find the vertex I want to change, find the closest features of the primitive that I want to change. If you wonder how to do that, then it is a different question.