Two ways of thinking of how LoRA applies a linear transformation to a d-vector:

1. compressing the d-vector into an r-vector by multiplying it by a d x r matrix, and then expanding the r-vector back to a d-vector by multiplying it by an r x d matrix.

2. multiplying the d-vector by a d x d matrix which is an approximation of an arbitrary d x d matrix (i.e., an arbitrary linear transformation), where the arbitrary d x d matrix is approximated as the product of a d x r matrix and an r x d matrix.

Computationally speaking, you'd want to do the first, but the two are equivalent by associativity of matrix multiplication.

Sources for images:
ninjalabo.ai/blogs/Low_Rank_Approximation-Part1.html
www.youtube.com/watch?v=DhRoTONcyZE