Quartz Is Not X11 (But It Could Be)
When OS X was first released, there was a long post by one of the Quartz developers on Slashdot explaining why they chose not to use X11. The argument was quite long, but it basically boiled down to this fact: To get the features that they wanted, they'd have had to implement so many extensions that they'd basically be running an entirely new windowing system anyway, just using X extensions to communicate with it instead of their own system. Now, all of those extra extensions exist and are well-supported by X.org, so the decision might be somewhat different. That hasn't stopped Google from implementing its own display system with Android. The saying about those who fail to understand UNIX being condemned to reinvent it poorly is even truer when it comes to X11.
There are architectural similarities between Quartz and X11. More, in fact, than between Quartz and its predecessors at Apple and NeXT: QuickDraw and Display PostScript. QuickDraw, found on the old MacOS Classic systems, was a very simple design. One process mediated access to the display, allocating regions to other processes, and the other processes drew into these directly. The obvious disadvantage to this was that all processes had access to the frame buffer, so could corrupt the display easily and synchronizing drawing between applications was very difficult.
Display PostScript worked in a very different way. Processes on old NeXT systems sent PostScript programs to the display server, which then ran them. This had its own problems, such as the fact that the display server effectively needed to be a complex multitasking virtual machine to prevent programs from taking over all of the display's time.
The design of Quartz is conceptually much simpler. Just as a filesystem virtualizes the disk by providing virtual disks (files) to applications, Quartz virtualizes the screen by providing virtual frame buffers (windows) to applications. I should mention that Apple's documentation often calls this part of Quartz the “Quartz Compositor” or simply the “Window Server,” and includes a few other things (including Core Animation) in the Quartz brand. The window server provides a region of shared memory to clients for drawing and then composites this into the frame buffer (on the GPU). It's a little bit more complicated than that, because this buffer is now an OpenGL texture and can be the result of rendering OpenGL graphics to the texture or compositing other textures together.
So what advantages did Quartz have over X11? Over a modern X11 system, none, and several disadvantages that come from tying the policy into the window server. These days, the window server is implemented on top of OpenGL, and the 2D drawing is handled either in software or by sending OpenGL commands to the window server, exactly as would happen with X11. Unlike X11, the policy for compositing and managing windows is implemented entirely in the window server. With X11, this is handled by a separate processes (or two; the window manager and compositing manager, which are sometimes combined into a compositing window manager), and so you can use the same mechanisms on a handheld device or a multi-monitor workstation, just with different policies.
Many improvements have been made over the kind of X11 implementation that was present when Quartz was introduced. Back then, X didn't have any kind of compositing model. There was no sensible support for buffering. Applications each had a region of the display reserved on their behalf, as with QuickDraw, but drew into it with commands sent from the client. If one window covered another, then the data from the lower window was lost and needed redrawing later. Font rendering was a mess. Quartz fixed all of these and provided much of the inspiration from the later X extensions which served a similar purpose.
In terms of current capabilities, Quartz and a recent version of X.org are similar. The difference is that X.org has a cleaner separation of policy and mechanism, and maintains backwards compatibility with applications dating back to the mid 1980s. Because the communication layer between the Quartz frameworks and the window server are private, it would be quite easy to replace them with X.org (although this wouldn't gain Apple much), and X.org itself has been ported to run on top of Quartz in the form of Apple's X11.app. At the time of its creation, Quartz was clearly superior to X.org. Now they are roughly equivalent in terms of features, and Quartz is closely integrated with other parts of the system.