No doubt you have read other posts where folks have broken the arms/tabs that hold the memory in place if they are pushed too far out.
I use a small screwdriver, such as a jeweler's screwdriver, and have had no issues removing the memory multiple times.
With the screwdriver, I usually push the left arm out slightly. I do the same on the right side, and the memory will pop up so that you can remove it. I push on the arms - outward - right where the arm tab sets into the memory card. It should not take much effort and the arms do not have to go out much at all.
Hope this helps ...