The object format is selected on the yasm(1) command line by use of the -f objfmt command line option.
The "bin" object format produces a flat-format, non-relocatable binary file. It is appropriate for producing DOS .COM executables or things like boot blocks. It supports only 3 sections and those sections are written in a predefined order to the output file.
The COFF object format is an older relocatable object format used on older Unix and compatible systems, and also (more recently) on the DJGPP development system for DOS.
The "dbg" object format is not a "real" object format; the output file it creates simply describes the sequence of calls made to it by Yasm and the final object and symbol table information in a human-readable text format (that in a normal object format would get processed into that object format's particular binary representation). This object format is not intended for real use, but rather for debugging Yasm's internals.
The ELF object format really comes in three flavors: "elf32" (for 32-bit targets), "elf64" (for 64-bit targets and "elfx32" (for x32 targets). ELF is a standard object format in common use on modern Unix and compatible systems (e.g. Linux, FreeBSD). ELF has complex support for relocatable and shared objects.
The Mach-O object format really comes in two flavors: "macho32" (for 32-bit targets) and "macho64" (for 64-bit targets). Mach-O is used as the object format on MacOS X. As Yasm currently only supports x86 and AMD64 instruction sets, it can only generate Mach-O objects for Intel-based Macs.
The RDOFF2 object format is a simple multi-section format originally designed for NASM. It supports segment references but not WRT references. It was designed primarily for simplicity and has minimalistic headers for ease of loading and linking. A complete toolchain (linker, librarian, and loader) is distributed with NASM.
The Win32 object format produces object files compatible with Microsoft compilers (such as Visual C++) that target the 32-bit x86 Windows platform. The object format itself is an extended version of COFF.
The Win64 object format produces object files compatible with Microsoft compilers that target the 64-bit "x64" Windows platform. This format is very similar to the win32 object format, but produces 64-bit objects.
The XDF object format is essentially a simplified version of COFF. It's a multi-section relocatable format that supports 64-bit physical and virtual addresses.
Peter Johnson <peter@tortall.net>