Servers with local disks keep all data on a local RAID mirror on the host system. They are optimized for high I/O performance and low latency and are especially suited for applications which require fast access to disks with low latency, such as databases.
Servers with ceph disks store their data on a remote filesystem. Each block is stored on three different servers. They are especially suited for higher availability needs: If the local host hardware fails, we will boot the server on a different machine.