Gems can be found by whacking bushes and critters, or rarely given as quest rewards.
There are ten types of Gem total:
- Agate; Aquamarine; Citrine; Topaz; Sapphire; Ruby; Jade; Moonstone; Emerald; Diamond.
Markus the Gem Trader in Ye Olde Shoppe in Haven - the NPC behind the counter on the left - will let you exchange gems both to upgrade and downgrade the types. There is no cost involved whatsoever, nor any penalty for switching gems back and forth.
Look for the Gem Type you want to trade in the table below, then find what you want to trade it into to see how many you need to upgrade, or how many you'll receive if you downgrade.
If you need it written out though, try this:
To turn an Agate (the basic gem) into an Aquamarine (the gem one tier higher) you'll need 4 Agate per Aquamarine. And if you so wish, you can turn the Aquamarine back into 4 Agates with no penalty.
If you wish to turn your Agate into Citrine, the math works out at x4, x4, to a total of 16 Agate for a single Citrine - in other words, there's an exponential growth in how many low tier gems are required to gain the higher and higher tier gems.
Using only Agate, if you were to try to gain a single Diamond you'd need a grand total of 262,144!
Fortunately, the deeper into the game you get the better the gems you gain as you whack away.