Hooking Directx Endscene from an Injected Dll

Hooking DirectX EndScene from an injected DLL

You install a system wide hook. (SetWindowsHookEx) With this done, you get to be loaded into every process.

Now when the hook is called, you look for a loaded d3d9.dll.

If one is loaded, you create a temporary D3D9 object, and walk the vtable to get the address of the EndScene method.

Then you can patch the EndScene call, with your own method. (Replace the first instruction in EndScene by a call to your method.

When you are done, you have to patch the call back, to call the original EndScene method. And then reinstall your patch.

This is the way FRAPS does it. (Link)

---

You can find a function address from the vtable of an interface.

So you can do the following (Pseudo-Code):

IDirect3DDevice9* pTempDev = ...;
const int EndSceneIndex = 26 (?);

typedef HRESULT (IDirect3DDevice9::* EndSceneFunc)( void );

BYTE* pVtable = reinterpret_cast<void*>( pTempDev );
EndSceneFunc = pVtable + sizeof(void*) * EndSceneIndex;

EndSceneFunc does now contain a pointer to the function itself. We can now either patch all call-sites or we can patch the function itself.

Beware that this all depends on the knowledge of the implementation of COM-Interfaces in Windows. But this works on all windows versions (either 32 or 64, not both at the same time).

EndScene hook questions

How do you know which functions you need to hook?

To put it bluntly, you have to be an experienced DirectX graphics programmer to find that out. Don't expect being able to hook into a framework that you don't understand. It just so happens that EndScene will always be called after all the other draw calls on the render target.

There are tons of D3D9 programming resources available, online and in paper form. Most of them are not free. I'm afraid this is not the answer you were hoping for.

What is the deal with pattern scanning, or creating a temporary D3D9 device?

Microsoft did not put any explicit effort into making EndScene hookable. It just happens to be hookable because every normal function is hookable. You need a way to find the function in memory, because the function will not always be at the same address.

One approach is to scan for known instructions that appear inside the function. Someone needs to be the first person to find out that pattern that you can scan for. You are far from the first person to hook EndScene, so many have reverse-engineered the function before and shared searchable patterns.

NOTE: The pattern does not necessarily need to be directly inside the target function. It might also lead you to something else first, in your case, the ID3D9Device instance. The important thing is that you can find your way to the EndScene function somehow.

Another approach is to get a pointer to the function. If it was a regular C function, that would be easy. It's hard here because OOP tends to make these things hard - you have to fight your way through various interfaces to get the correct vtable.

Both methods have advantages and disadvantages -- creating a D3D9 device is safer, but also more intrusive, because the target process might not expect someone to just randomly create new devices.

Why does the hook function need __stdcall?

Since you replace the original function with your hooked version, the calling convention of the hooked function must be the same as the calling convention of the original function. The caller of EndScene expects (and was compiled with) a __stdcall convention, so the new function must also behave the same way, otherwise the stack will be corrupted. Your act of replacing the function does not change the way the caller calls it.

Direct2D Error NO_HARDWARE_DEVICE When Injecting Dll Into Another Process

Put directly the D2D creating function into new thread by CreateThread in DllMain.

Unloading an Injected DLL

I found the cause of the problem and the solution. I honestly feel quite stupid for missing it and struggling with it for so long.

As I mentioned in the original problem, the processes that are problematic do not have a UI. Turns out they do have a message pump running. The problem is nothing is sending messages to these processes without a UI after we call UnhookWindowsHookEx that would trigger the unloading. (In fact, I believe MSDN does state that window messages are not sent to processes when calling UnhookWindowsHookEx.)

By broadcasting WM_NULL to all processes after the injecting process calls UnhookWindowsHookEx the message pump wakes up in the injected processes and the DLL reference count is decremented. The DLL is unloaded immediately when the injected DLL finally calls FreeLibraryAndExitThread.

This is only part of the solution. If the injecting process is killed or crashes, no message is broadcasted so the DLL does not get unloaded from processes that do not have a UI. As I mention before I have a thread running in the DLL that waits on the injecting process handle. When the injecting process ends the DLL is signaled and then calls PostThreadMessage to send WM_NULL to each thread in the process. Then it waits until the DLL reference count is decremented before continuing and cleaning up before calling FreeLibraryAndExitThread. As a result, the DLL is unloaded almost immediately from all processes, UI or no UI.

---

Related Topics