Data Execution Prevention (DEP) causes LoadLibrary to fail

MSCPP Ever came across an “Invalid access to memory location” when accessing a DLL by LoadLibrary? Well apparently the Data Execution Prevention, added with Service Pack 3 of Windows XP, steps in if the Characteristics field of the section header in that DLL is missing the IMAGE_SCN_MEM_EXECUTE attribute. If the DLL’s source code is not available one would need a tool to modify the DLL – or disable DEP for the current process with SetProcessDEPPolicy(0). The latter can be accomplished in this way:

SML::HModuleDLLRef kernel32DllHandle(LoadLibrary(L"kernel32.dll"));
boost::function<BOOL (HANDLE, DWORD*, BOOL*)> GetProcessDEPPolicy
 = (BOOL (WINAPI*)(HANDLE, DWORD*, BOOL*)) GetProcAddress(
   kernel32DllHandle.getHandle(), "GetProcessDEPPolicy");
boost::function<BOOL (DWORD)> SetProcessDEPPolicy
 = (BOOL (WINAPI*)(DWORD))GetProcAddress(
   kernel32DllHandle.getHandle(), "SetProcessDEPPolicy");

if (!GetProcessDEPPolicy || !SetProcessDEPPolicy)
 THROW0(Win32, ::GetLastError(), L"'kernel32.dll' error!");

DWORD                               DEPFlags;
BOOL                                permanent, fnSuccess;
fnSuccess = GetProcessDEPPolicy(::GetCurrentProcess(), &DEPFlags,
   &permanent);
if (!fnSuccess)
 THROW0(Win32, ::GetLastError(), L"Could not query DEP informations");

if ((permanent == FALSE) && (DEPFlags != 0))
{
 /// Disable DEP for this process otherwise LoadLibrary will fail
 fnSuccess = SetProcessDEPPolicy(0);
}

Note, TRACE and THROW are my helper macros and in the namespace SML reside my private Win32-API wrapper classes.

Tags: ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: