Difference Between Console (/Subsystem:Console) and Windows (/Subsystem:Windows)

difference between Console (/SUBSYSTEM:CONSOLE) and Windows (/SUBSYSTEM:WINDOWS)

/SUBSYSTEM:CONSOLE) is for console based applications. You should define main function in code.

/SUBSYSTEM:WINDOWS) is for GUI applications. You should define WinMain function.

Effects of switching between /SUBSYSTEM:CONSOLE to /SUBSYSTEM:WINDOWS in a DLL

This option only has an effect on applications (exe), not on libraries(dll).

Its documentation says:

The /SUBSYSTEM option specifies the environment for the executable.

The choice of subsystem affects the entry point symbol (or entry point
function) that the linker will select.

This does not affect libraries, which have their own (optional) entry point.

Additional information following CyberShadow's comment: The subsystem field itself seems to be ignored when a DLL is loaded. This article about CSRSS says (emphasis mine):

Furthermore, each process is associated with one, certain subsystem;
this property is being set by the linker (during the compilation
process), and resides in the following PE structure field: [...]

how to fix setting subsystem to console in visual studio?

I had the same problem. The only way to resolve this issue was to create a custom project template in Visual Studio. Refer this link on how to create a project template in Visual Studio

Decide an application to be of console/windows subsystem at run time

I'd suggest to keep your GUI application with the windows subsystem.

At the very beginning, when parsing command line, instead of creating the GUI windows (MFC inistialisation), you could create a console or attach with AttachConsole()

As a proof of concept, here how it could look like (for example in CxxxApp::InitInstance()):

...  // init code but before the the CMainFrame is created 

if(...) {   // suppose you want to go to the console
    if (! AttachConsole(ATTACH_PARENT_PROCESS))   // try to hijack existing console of command line
        AllocConsole();                           // or create your own.

    DWORD nw,nr;    // demo with windows native console i/o 
    char buff[32]; 
    WriteConsole(GetStdHandle(STD_OUTPUT_HANDLE), L"Who are you ? ", 14, &nw, NULL); 
    ReadConsoleA(GetStdHandle(STD_INPUT_HANDLE), buff, sizeof(buff), &nr, NULL);
    WriteConsoleA(GetStdHandle(STD_OUTPUT_HANDLE), "Hello ", 6, &nw, NULL);
    WriteConsoleA(GetStdHandle(STD_OUTPUT_HANDLE), buff, nr, &nw, NULL);
    ReadConsoleA(GetStdHandle(STD_INPUT_HANDLE), buff, 1, &nr, NULL);
    ...
}

If you want to use C++ cin/cout, additional work is however needed. This other SO question addresses for example the redirecting of cout.

How can I specify the subsystem in the CodeBlocks additional linker options?

Passing -Wl,--subsystem,windows to the linker is the proper way to do this. If it appears not to work then make sure that you didn't insert a space somewhere; use the argument as it is.

Console Application Console window - Visual C++

In the Project Properties on the Configuration Properties->Linker->System page, you need to set the value of SubSystem to Windows (/SUBSYSTEM:WINDOWS). The default for a new console application project is Console (/SUBSYSTEM:CONSOLE), which causes Windows to allocate a new console or attach to the parent process's console when starting your program.

---

You also need to change your main function to be WinMain. The signature for `WinMain is:

int CALLBACK WinMain(
  _In_ HINSTANCE hInstance,
  _In_ HINSTANCE hPrevInstance,
  _In_ LPSTR     lpCmdLine,
  _In_ int       nCmdShow)
{
    // Your code here
}

---

With the above approach, child console processes will still create console windows. Since you stated in a comment that you want to use popen, you can't really easily use the normal way of calling CreateProcess with SW_HIDE.

What you really want to do is to attach a hidden console window to your process and allow your child processes to inherit it. This probably isn't the best code, but here's a way to do it:

// Allocates a hidden console window for this process. This console can be
// inherited by child console processes, preventing them from creating a
// visible console. Returns false if the attempt fails.
bool AllocHiddenConsole()
{
  TCHAR command[] = _T("cmd.exe");
  STARTUPINFO startupInfo{};
  PROCESS_INFORMATION processInfo{};
  startupInfo.cb = sizeof(startupInfo);
  startupInfo.dwFlags = STARTF_USESHOWWINDOW;
  startupInfo.wShowWindow = SW_HIDE;

  if (!CreateProcess(NULL, command, NULL, NULL, FALSE,
    CREATE_NEW_CONSOLE, NULL, NULL, &startupInfo, &processInfo))
  {
    return false;
  }

  bool attached = false;
  for (int i = 0; i < 1000; i++)
  {
    if (AttachConsole(processInfo.dwProcessId))
    {
      attached = true;
      break;
    }
    Sleep(10);
  }

  TerminateProcess(processInfo.hProcess, 0);
  CloseHandle(processInfo.hProcess);
  CloseHandle(processInfo.hThread);

  return attached;
}

Running NI-IMAQ and WINAPI requires 2 different SubSystems

There can only be one subsystem in place. As a window is meant to be displayed, use the Windows subsystem.

Between int main() and int WINAPI WinMain(), use int WINAPI WinMain() as the main function and include the function calls from int main().

Difference between Windows and Console application

The sole difference is that a console application always spawns a console if it isn't started from one (or the console is actively suppressed on startup). A windows application, on the other hand, does not spawn a console. It can still attach to an existant console or create a new one using AllocConsole.

This makes Windows applications better suited for GUI applications or background applications because you usually don't want to have a terminal window created for those.

On a more technical note, the only difference between a Console and a Windows executable is one byte in the PE header of the exe file. Toggling this byte manually (e.g. using a hex editor) converts the application type. This is a well-published hack that is used to create console applications in VB6 (where this type of application was not explicitly supported).

To determine and change the subsystem type of an application, you need to read parts of the PE header. The address of the subsystem data is not fixed though, because it's part of the optional file header whose position is determined by an address stored in the DOS file header (in the member e_lfanew). This address actually points to the _IMAGE_NT_HEADERS record which, in turn, includes the IMAGE_OPTIONAL_HEADER32 structure. This has an int16¹⁾ member called Subsystem. The member's value is 2 for a Windows application and 3 for a console application. Other subsystems exist (in particular, POSIX and kernel).
I've written a small VB6 application to change the subsystem of an application, which can be downloaded from ActiveVB as source code.

The PE format isn't very well documented but this document may serve as an introduction: Peering Inside the PE: A Tour of the Win32 Portable Executable File Format.

---

¹⁾ This doesn't really contradict my claim that only one byte differs: the most significant byte of this member is always 0. Only the least significant byte changes.

---

Related Topics