- CPU(s) - Multi-threaded request fulfillment
- RAM - To do map composition, reprojection, format conversion
- Database - To draw spatial data out of, vector data greatly benefits from being in the database and being indexed
- HardDrive/File Share Service - Raster data is best drawn out of stripe-raid drives (0,5,0+1 depending on your budget).
PS: Warning before touching anything (thanks arneke).