Intercepting messages inside Is­Dialog­Message, installing the message filter

Last time, we saw that one way to intercept the ESC in the standard dialog message loop is to use your own dialog message loop. However, you might not be able to do this, say, because the dialog procedure uses End­Dialog() , and the dialog exit code is not retrievable from a custom message loop.

The Is­Dialog­Message includes an extensibility point that lets you hook into the message processing. You can register a message filter hook and listen for MSGF_ DIALOG­BOX .

Before processing a message, the Is­Dialog­Message function does a Call­Msg­Filter with the message that it is about to process and the filter code MSGF_ DIALOG­BOX . If the filter result is nonzero (indicating that one of the hooks wanted to block default processing), then the Is­Dialog­Message returns without doing anything. This lets us grab the ESC from Is­Dialog­Message before it turns into an IDCANCEL .

Here’s our first attempt. (There will be more than one.)

HWND hdlgHook; #define DM_ESCPRESSED (WM_USER + 100) LRESULT CALLBACK DialogEscHookProc(int nCode, WPARAM wParam, LPARAM lParam) { if (code == MSGF_DIALOGBOX) { auto msg = (MSG*)lParam; if (IsDialogESC(hdlgHook, msg)) { return SendMessage(hdlg, DM_ESCPRESSED, 0, lParam); } } return CallNextHookEx(nullptr, nCode, wParam, lParam); }

Our hook procedure first checks that it’s being called by Is­Dialog­Message . if so, and the message is a press of the ESC key destined for our dialog box (or a control on that dialog box), then send the dialog box a DM_ ESC­PRESSED message to ask it what it thinks. The dialog procedure can return TRUE to block default processing or FALSE to allow default processing to continue.

Here is the handler in the dialog procedure itself:

INT_PTR CALLBACK DialogProc(HWND hdlg, UINT message, WPARAM wParam, LPARAM lParam) { switch (message) { case WM_INITDIALOG: hdlgHook = hdlg; ⟦ other dialog initialization as before ⟧ ⟦ ending with "return (whatever)" ⟧ case DM_ESCPRESSED: if (⟦ we want to process the ESC key ourselves ⟧) { ⟦ do custom ESC key processing ⟧ SetWindowLongPtr(hdlg, DWLP_MSGRESULT, TRUE); return TRUE; } break; ⟦ handle other messages ⟧ } return FALSE; }

When the dialog initializes, remember its handle as the dialog for which the Dialog­Esc­Hook­Proc is operating.

When the dialog is informed that the ESC key was pressed, we decide whether we want to process the ESC key ourselves. If so, then we do that custom processing and set up to return TRUE from the window procedure. For dialog procedures, this is done by setting the message result to the desired window procedure result and then returning TRUE to block default dialog box message processing and instead return the value we set (which is TRUE ) from the window procedure.

... continue reading