Building a Window Manager

One important decision is how you're going to talk to the X server. You can use the Xlib bindings for your language of choice, or you can use the higher-level XCB bindings. (If you're insane, you might open a socket to the X server directly.)

To know how a window manager ought to behave, there are two documents that specify the conventions and policies: EWMH and ICCCM¹. Conforming to these means your window manager will behave nicely in GNOME, KDE, XFCE, and any other desktop environment that comes along, although simply ignoring them is certainly easier on your first try.

A window manager needn't be a huge, complicated ball of C — Successful window managers have been written in high-level languages like Lisp, Haskell, and Python, and even some in C have remained small and readable. XMonad, written in Haskell, stayed under 1000 lines for quite some time. StumpWM (Common Lisp) and DWM (C) are both quite minimalist. You might be able to read their source code to get some inspiration as to how to design a WM.

¹ Elijah Newren wrote:

DO NOT GO AND READ THOSE THINGS. THEY ARE REALLY, REALLY BORING. If you do, you'll probably end up catching up on your sleep instead of hacking on Metacity. ;-)

Come to think of it, Metacity's documentation has a good bit to say about how it interacts with windows and what sort of extended properties it supports.

Modern ways to write a window manager

This is a late answer so maybe it doesn't have much relevance anymore but I'll post it anyway since other people might be looking for it...

I have asked myself the same question and came to the conclusion that there exists no such thing, especially not in Java. While certain widget toolkits, like qt, offer you the possibility to use an existing display struct and create widgets out of existing xlib window handles, you'll still have to use xlib/xcb to do the nasty job of communicating with the X server for wm specific operations. Other libraries, as stated by others, like GDK and the enlightenment also offer a pretty good job of basic abstraction.

Still, I wasn't happy with the fact that all of these use C/C++ and sitting on my lazy ass waiting until somebody did the work for me in Java wasn't an option, I started writing my own library that allows you to build your own modular WM, in Java.
https://github.com/Zubnix/trinityshell

Building a 3D window manager for Linux

Are there any window managers that meet these criteria and are lightweight enough for me to work with?

you could look at Compiz which does just that. Also it has a plugin architecture. However beware the pitfalls of X11 based compositing, the woes of X11 itself and that large portions of the Linux graphics community have drunken the Wayland Kool-Aid.

How to get started writing a compositing WM?

There are two parts in your question: 1) How to write WM 2) how to write composite manager

Some links to help understand part two (in addition to xcompmgr source):

http://www.talisman.org/~erlkonig/misc/x11-composite-tutorial/ (Uses Qt, but very generic and low level)
https://github.com/gustavosbarreto/compmgr
http://projects.mini-dweeb.org/projects/unagi

Window manager, "part one":

I have simple ~100 loc wm in JavaScript: https://github.com/sidorares/node-x11/blob/master/examples/windowmanager/wm.js
another minimalist wm (in C), good to start as a reference: https://code.google.com/p/winmalist/
most important keyword: SubstructureRedirect event mask. A bit of documentation here

How to design window manager in such a way that we can touch the home screen or interact with the UI behind that window manager

This is not possible, short of building your own custom ROM with your own custom Android build.

Generally speaking, touch events are delivered to the window highest on the Z axis at the X/Y coordinate of the touch event. If your window is highest on the Z axis, you get the touch events.

What is possible in theory, but not available to app developers, is to have some sort of API that "punches holes" in your window. In those holes, the touch events would not go to your window, but rather to whatever is next-highest on the Z axis at that X/Y coordinate. This would require some OS modification.

If, however, you are seeking some way that you can see the touch events, then dispatch them yourself to the underlying windows, that is blocked for obvious privacy and security reasons.

Writing Tiling window manager in Python

You will need some X client library. I suggest having a look at python-xlib, a pure Python implementation of the client side of the X protocol. It includes plwm, an example implementation of a minimal window manager written in Python.

Building a Window Manager