MPC + headphones and a bunch of samples is about a simple as it gets.
You'll no doubt wish to sample sounds into your MPC, so you'll need sound sources such as CDs, vinyl (and therefore a turntable), voice, acoustic instruments (via a mic) etc.
Amplifier and monitors (speakers).
Beyond that, you can take it as far as you want. Most people will utilise a computer, e.g. to run VST instruments via MIDI, or with a DAW to record and mix down their music - so you'll need a decent audio/MIDI interface. Then there is the option of using hardware such as synths, sound modules, external effects and processors.
As you progress you'll need to set up your room more efficiently with dampening, bass traps etc.
But if you want to keep it simple, MPC, headphones, amp + speakers and various sound sources.
