Why Does a Binary of One Os (Windows) Does Not Run in Other ( Linux) for Same Underlying Architecture

Why an executable program for a specific CPU does not work on Linux and Windows?

There are many differences. Among them:

Executable Format: Every OS requires the binaries to conform to a specific binary format. For Windows, this is Portable Executable (PE) format. For Linux, it's ELF most of the time (it supports other types too).
Application Binary Interface: Each OS defines a set of primary system functions and the way a program calls them. This is fundamentally different between Linux and Windows. While the instructions that compute 2 + 2 are identical on Linux and Windows in x86 architecture, the way the application starts, the way it prints out the output, and the way it exits differs between the operating systems.

Yes, both Linux and Windows programs on x86 architecture use the instruction set that the CPU supports which is defined by Intel.

Why are Executable files operating system dependent?

In order to do something meaningful, applications will need to interface with the OS. Since system calls and user-space infrastructure look fundamentally different on Windows and Unix/Linux, having different formats for executable programs is the smallest trouble. It's the program logic that would need to be changed.

(You might argue that this is meaningless if you have a program that solely depends on standardized components, for example the C runtime library. This is theoretically true - but irrelevant for most applications since they are forced to use OS-dependent stuff).

The other differences between Windows PE (EXE,DLL,..) files and Linux ELF binaries are related to the different image loaders and some design characteristics of both OSs. For example on Linux a separate program is used to resolve external library imports while this functionality is built-in on Windows. Another example: Linux shared libraries function differently than DLLs on Windows. Not to mention that both formats are optimized to enable the respective OS kernels to load programs as quick as possible.

Emulators like Wine try to fill the gap (and actually prove that the biggest problem is not the binary format but rather the OS interface!).

Why do we need to compile for different platforms (e.g. Windows/Linux)?

Even though CPU is the same, there are still many differences:

Different executable formats.
Different calling conventions might be used. For example Windows x64 passes integer args in different registers than the x86-64 System V ABI and has several other significant differences, including call-preserved xmm6..15 in Windows, unlike other x86-64.
Different conventions regarding stack structure. Some systems have a concept of "red zone" to help compiler generate shorter code. Execution environment has to honor such concept to avoid stack corruption.
Programs are linked against different standard libraries with different ABIs - field order might differ, additional extension fields might be present.
In both C and C++ some data types have OS dependent sizes. For example on x86_64 long is 8 byte on Linux, but 4 bytes on Windows. (Type sizes and required alignments are another part of what makes an ABI, along with struct/class layout rules.)
Standard libraries can provide different set of functions. On Linux libc provide functions like snprintf directly, but on Windows snprintf might be implemented as static inline function in a header file that actually calls another function from C runtime. This is transparent for programmer, but generates different import list for executable.
Programs interact with OS in a different way: on Linux program might do system call directly as those are documented and are a part of provided interface, while on Windows they are not documented and programs should instead use provided functions.
Even if two OS rely on program doing system calls directly, each kernel has its own set of available system calls.

Even if a Linux program only calls the C library's wrapper functions, a Windows C library wouldn't have POSIX functions like read(), ioctl(), and mmap. Conversely, a Windows program might call VirtualAlloc which isn't available on Linux. (But programs that use OS-specific system calls, not just ISO C/C++ functions, aren't portable even at a source level; they need #ifdef to use Windows system calls only on Windows.)

Not OS related, but programs compiled by different compilers might not be interoperable: different standard libraries might be used, things like C++ name mangling might be different, making it impossible to link libraries against each other, C++ exception implementation might be non-interoperable.
Different filesystem structure. Not only there is a difference between "" on Windows and "/" on Unix-likes, but there are "special files" that might or might not be present like "/dev/null".

In theory everything listed here can be resolved: custom loaders can be written to support different executable formats, different conventions and interfaces do not cause problems if the whole program uses the same set of them. This is why projects like Wine can run Windows binaries on Linux. The problem is that Wine has to emulate functionality of Windows NT kernel on top of what other OSes provide, making implementation less efficient. Such program also have problems interacting with native programs as different non-interoperable interfaces are used.

Source-compatibility layers like Cygwin can be inefficient, too, when emulating POSIX system calls like fork() on top of the Windows model. But in general Cygwin has an easier job than WINE: programs need to be recompiled under Cygwin. It doesn't try to run native Linux binaries under Windows.

Machine Independency

The binary setup of the object files are totally different. Also which libraries are available or how to call them.

Just compare the header of an ELF or an EXE file to see what I mean.

If you write a simple program like "main(){printf("Hello\n"); return 0;} there is a lot going on behind the scenes that are covered by the compiler to get these lines printed. Running on the same CPU doesn't help, because it could execute the assembly instructions, but it would fail horribly as soon as calling the first OS function.

To elaborate this a bit:

Just as an exmaple. Lets assume that we are running on Amiga OS with a Motorola 68000 CPU.

If I remember correctly, the calling convetions to call a system library involved loading the pointers into i.e. an adress register of the CPU and then call the OS function.

Now lets assume I write my own OS also using a Motorola 68000 CPU. However, when I design my OS, I thought it is a much better idea to use the stack for data exchange, so when you call a similar function in my own private OS, you don't pass the adress in the address register, instead you push it on the stack.
Now when your executable would be executed in my OS (supposing it could be loaded because I use the same object structure) your executable would put values in a register and my OS would try to pop them from the stack, because it doesn't know that the values it was looking for were supposed to be somehwere else.

I hope this is a bit more detailed so you can understand it, but of course the problems go much deeper then this, as this is just a tiny part of the problems involved.

Binary compatibility over what range of machines?

There are several levels/sources of binary incompatibility.

Firstly, addressing library incompatibility

Normally, if you're running a binary on another machine, with a different version of the "same OS" (whatever that means...), then it will run fine. Completely. The issue isn't with code not working, but missing code: the bits of the OS that the binary depends on that aren't there on the target machine. It's a failure to find code to run, not to run code (which would be perfectly OK, until the point where it tried to use the missing bit!)

So, binaries from your Ubuntu gcc will most likely run on any linux system that's no older than the machine it was compiled on. It depends on exactly what functionality the binary depends on from the OS and system libraries.

Very few binaries have no external dependencies. Examine the dependencies using ldd on your output. The one most likely to cause problems is the libgcc dependency. libc and friends change very infrequently, so hardly ever cause difficulties with compatibility. GCC changes often, so will restrict the target machines your binaries will run on.

The general rule is: use as your build machine the oldest distro you want to run on. So, with a RHEL 2 build machine, you'll be able to run the binary on any system no older (with some exceptions). That's a common guideline that keeps things simple. If you need binary compatibility across lots of distros, statically linking to libstdc++ (if you use C++) is an excellent choice. Statically linking to libgcc is dangerous, and not to be attempted unless you really know what you're up to.

Finally, note that library compatibility is much simpler on other UNIX platforms; it's only linux that's such a pain. Anything compiled on AIX 6 (say) or SunOS 5.8 or HP-UX 11.00 or whatever runs with no issues on all later releases. The environment is homogenous enough that they can just ship with piles of legacy cruft in their system libraries, to guarantee that every symbol present on the old release is available on the latest one, with the same semantics.

Secondly, cross-OS binaries: run-time differences between OSes

It's all just machine code, so you might think binaries should work on other OSes. They won't. A big reason is system calls: when you need to invoke something from the kernel (which is necessary for most non-trivial functionality), you have to know how to "talk to the kernel". That is, you make a syscall and tell the OS, "do the 42 thing, you know".

The numbers for the syscalls depend on the kernel. So, the Solaris binary would be OK from linux, apart from that niggle that stops pretty much everything working.

In fact, some OSes do support the syscalls of other kernels. FreeBSD knows the linux syscalls, and when you tell it "do the 42 thing", it checks a flag in the header of your ELF, and does the appropriate action for the linux syscall numbers if the ELF is stamped as a linux one. Neat. (Of course, you need to link in linux libs as well... But, a statically linked linux binary will run fine in FreeBSD with no externals.)

Thirdly, load-time cross-OS compatibility

A binary isn't "just a binary"; it contains a whole load of meta-data that gets interpreted by the kernel before it can run the thing. Most of the binary is a blob of code and data, plus linker information. But, even something that doesn't have any external dependencies has to be parseable by the kernel.

Some kernels understand multiple binary formats, such as PE, ELF, or a.out (obsolete). The linux binaries will never run on Windows, even if they don't make any syscalls (such a binary can't exit cleanly, but for the sake of example...). That's because MS is not about to add ELF support to their kernel: the wrapper around the code that describes how it is to be loaded and started can't be read by Windows, so the code inside it won't be run: Windows doesn't know which bits of the file are even code!