A rootkit is a piece of malicious software which has the advanced capability of hiding itself on an infected system. This is usually done by hooking system functions. For example a rootkit could be used to hide files from the user by hooking functions responsible for listing the contents of a directory. Rootkits are frequently used in combination with other malware, which it hides from users and security products. The number of malware families that have rootkit capabilities and targeting Microsoft's Windows systems is well into double figures.
We think that there could be rootkits targeting the OS X platform, but we have very limited visibility into that threat right now. We know that we don't know. We also know that various websites and even paperback books [1, 2] document how rootkits can work under OS X. We have seen OS X malware using rootkit techniques in the past. The most notable example being OSX/Morcut [3] also called Crisis by other vendors. This malware was used to steal information from infected Macs and loaded a kernel extension so as to hide its files from the victim.
Detecting a rootkit under OS X currently involves dumping and analyzing kernel memory. It requires time and knowledge. It is not something accessible to everyone.
via Known unknowns - detecting rootkits under OS X - We Live Security.