The Xbox's Original Software is composed of many things, including the bootloaders, the kernel, the dashboard and more.
Bootloaders
The Xbox 360 system uses a series of bootloaders to load the hv/kernel and dashboard. These loaders are explained at Boot Process.
Hypervisor
The Hypervisor is the main operating system being run on the console. It looks after the kernel and stops things like buffer overflows from happening. It also manages a lot of the security aspects of the 360, including memory encryption and region protection. It has code to handle the XeKeys (keyvault + some static ones) and XEXs. The end-user of an Xbox would never have any knowledge of the hypervisor, it's only really interesting for hackers.
Kernel
The Xbox 360's Kernel is responsible for low level hardware communication and more. This is the main part of the 360, the kernel is where all the user-based stuff happens. The drivers for accessing most of the hardware is located in the Kernel, along with certain drivers for other things used by the system (e.g. FATX (File Allocation Table for Xbox), STFS (Secure Transacted File System), SFCX (System Flash Controller for Xbox). This is the most interesting part of the console because of all the functions it performs.
You can check your kernel version in "System -> Console Settings -> System Info". The line at the bottom contains your kernel version in the format K:2.0.nnnnn.0, where nnnnn is your kernel version.
XAM
A lot of fun things are inside XAM. XAM handles gamer profiles and online interactivity, so as you can guess not a lot of information is made public about it. XeDevWiki hopes to change that. XAM handles such things as XDBF (Xbox Database File) files (GPD (Gamer Progress Data)/SPA) and PEC (Profile Embedded Content) files.
File Systems
The Xbox 360 uses various File Systems for storing games, user content and more.
FATX (File Allocation Table for Xbox) is used as the storage filesystem on hard drives/memory units/USB devices/xlaunch.fdf files.
GDFX/XSF (Game Disc Format for Xbox) is the filesystem used on X360 CD/DVD Media.
STFS (Secure Transacted File System) is used for game saves, profiles, arcade games, downloadable content and more. STFS is also referred to as CON/LIVE/PIRS files, as they are all STFS files, just signed with a different header.
(SFCX Secure File Cache for Xbox) :is used for cache storage for games.
NAND File System is used to store the bootloaders, kernel, keyvault and other files on the NAND.
File Formats
The Xbox 360 also uses different File Formats for storage of data, as storage on the Xbox is limited, these files are usually binary. XeNTaX.com is a DataBase of hundreds of different file formats, which may be of help.
PEC (Profile Embedded Content) is used as another layer of security inside profiles.
STFS (Secure Transacted File System) is used for secure storage of files on memory devices.
XDBF/GPD/SPA (Xbox Dashboard File/Gamer Profile Data/Statistics, Presence and Achievements) is used for profiles (achievements/settings/titles) and games (achievements/strings/ratings).
XEX (Xenon Executable) is the executable format used by the Xbox 360.
XMI(Xenon Music Index) is used to index music on the Xbox.