Here's the specification of the ISO base format, and here is the MP4 file format specification, both defined as part of the ISO/IEC 14496 standard.

Also take a look at the QuickTime MP4 specification. It has a very clear explanation.