I've been trying to create a Camera class that uses Quaternion and Vector3 to create a view matrix but I haven't been having trouble with the math. I'm hoping someone can direct me to a text book or some topics I can look up.

This is what I have right now:

class Camera
{
public:
    Camera();
    
	Vector3 Right() const { return mRight; }
	Vector3 Up() const { return mUp; }
	Vector3 Forward() const { return mForward; }

	Vector3 Position() const { return mPosition; }
	void SetPosition(const Vector3& position)
    { 
        mPosition = position; 
        Internal_Recalculate();
    }

	Quaternion Rotation() const { return mRotation; }
	void SetRotation(const Quaternion& rotation)
    { 
        mRotation = rotation;
        Internal_Recalculate();
    }

private:
    void Internal_Recalculate();

	Quaternion mRotation;
	Vector3 mPosition;

	Vector3 mRight;
	Vector3 mUp;
	Vector3 mForward;
};

Camera::Camera()
{
    mPosition = Vector3(0.0f, 0.0f, 0.0f);
    mRotation = Quaternion(0.0f, 0.0f, 0.0f, 1.0f);
}

void Camera::Internal_Recalculate()
{
    const Vector3 kForward = Vector3(0.0f, 0.0f, 1.0f);
    const Vector3 kUp = Vector3(0.0f, -1.0f, 0.0f);
    const Vector3 KRight = Vector3(1.0f, 0.0f, 0.0f);

    mForward = mRotation * kForward;
    mRight = mRotation * KRight;
    mUp = mRotation * kUp;
}

    float xRotate = camYRotRate * mGamePad.GetAxis(GamePadAxis::TmbRgtX) * -1 * dt;
    rotation = glm::rotate(rotation, xRotate, mCamera.Up());

    mCamera.SetRotation(rotation);

    float yRotate = camYRotRate * mGamePad.GetAxis(GamePadAxis::TmbRgtY) * dt;
    rotation = glm::rotate(rotation, yRotate, mCamera.Right());

    mCamera.SetRotation(rotation);

The first code segment is the camera class, and the second segment is the code I'm using to rotate the camera around. I'm just looking for any pointers, I've been stuck on this for weeks. I'm using the glm math library and directx 11

What problems are you seeing?

Off the bat, I see some issues though.

First, you better have a really good reason why you are using vectors and a quat for your camera instead of a matrix. A single matrix contains all of those "right, up, forward" vectors, plus it has position and rotation all baked together. And it can be rotated as a single unit. And it is easy to keep normalized and orthogonal.

It looks like you are trying to perform Euler rotations which do not behave quite how you'd expect. The order of operations matters and there is a big difference between world-space and local-space rotations.

I would recommend that you do the Y-rotation in world-space first and then the X-rotation in local space. In that order.

You are right for storing a local copy of your transforms for some backup purposes but it is also right that you in the end need a matrix to come along with your camera.

Taking a look at the basic shader you need in a graphics environment (this is GLSL but just to clarify)

uniform mat4 projection;
uniform mat4 view;
uniform mat4 model;

void main()
{
    gl_Position = projection * view * model * vert;
}

You see how a fragment is computed by multiply your projection (either otrhogonal or perspective normaly in a game) with the current view (where is the camera located, where does it look to) and the models "view" (where is the mesh located, where does it look to)

Based on this you anyway need a matrix for rendering out of your camera and it makes life easier when operating on your camera. It is inefficient to recompute rotation or transform from a matrix but storing a bit extra data is totally ok here. What you should do is to update your cameras internal matrix using a transform/rotation matrix out of a passed value (when doing recompute). Then you could realy simple transform your forward, up and right vectors based on your internal matrix

I concur, keep some vectors and stuff around for backup, but the camera itself should be a matrix.

Any 3d camera is really two matrices. One is the transform of the camera in the scene. The other is the projection matrix that is sent to the shader.

Here's my camera class, which is designed as a targeted "look-at" camera.

public class camera
{
  public Matrix4 world, projection = new Matrix4();
  public Vector4 position, target, up = new Vector4();
  public float aspect, fov, near, far;
}

The class has two matrices : the world matrix is the actual camera in the scene. In this case, it is generated as a "look-at" transform. The projection matrix is what is then sent to the shader.

There are three vectors : position is the position of the camera itself, target is the position of the camera's target. The up vector is used to for computing the world matrix (changing this value will "tilt" or "roll" the camera)

The four float values are all parameters used to compute the final projection matrix.

Every frame, this class executes an update method which recalculates the two matrices using the other properties. This makes animating the camera easy.

I concur, keep some vectors and stuff around for backup, but the camera itself should be a matrix.

Any 3d camera is really two matrices. One is the transform of the camera in the scene. The other is the projection matrix that is sent to the shader.

Here's my camera class, which is designed as a targeted "look-at" camera.

public class camera
{
  public Matrix4 world, projection = new Matrix4();
  public Vector4 position, target, up = new Vector4();
  public float aspect, fov, near, far;
}

The class has two matrices : the world matrix is the actual camera in the scene. In this case, it is generated as a "look-at" transform. The projection matrix is what is then sent to the shader.

There are three vectors : position is the position of the camera itself, target is the position of the camera's target. The up vector is used to for computing the world matrix (changing this value will "tilt" or "roll" the camera)

The four float values are all parameters used to compute the final projection matrix.

Every frame, this class executes an update method which recalculates the two matrices using the other properties. This makes animating the camera easy.

You can technically leave the camera at 0,0,0 and move everything around it too. At the end of the day the results are the same. Just saying :)

Thanks everyone, I switched my camera to use a matrix internally and the rotation problem I was having was related to me not using local and world axis correctly.