Why Certain Doctype Declarations Cause 100%-Height Tables and Divs to Stop Working

Why certain DOCTYPE declarations cause 100%-height tables and divs to stop working?

Because ancient browsers had odd, inconsistent behavior and browsers treat Doctypes like an intelligence test to see if the author is writing code to the standards or to what they learned from W3Schools a decade ago. If you have height: 100% and the height of the parent element is auto then 100% means auto.
Generally, you don't. It screams "layout table". That said, set heights or minimum heights on the html and body elements. There are other techniques, but I don't have a handy link at the moment as, oddly, I've never been in a position where I needed the technique.
It is what browsers are supposed to do, so …
Well, I am answering this question …

<!DOCTYPE html> prevents relative css heights

A doctype enforces a certain set of standards for the browser. If a page does not include a doctype, browsers will usually use some kind of quirks or transitional mode, which is more lenient about markup mistakes (but is bad practice and may display items incorrectly).

Essentially, strictly speaking, you can't set that element to height 100% using that browser's set of standards. It'll try to predict what you wanted to do if you don't include a doctype or include a transitional one and adjust the page's styling accordingly.

Why does height 100% work when DOCTYPE is removed?

CSS `height` property, percentage values & DOCTYPE

The first part of your question asking how to apply a 100% height to your div has been answered several times by others. Essentially, declare a height on the root element:

html { height: 100%; }

A complete explanation can be found here:

Working with the CSS height property and percentage values.

The second part of your question has received much less attention. I'll try to answer that.

Why does removing the doctype make [the green background] work?

When you remove the DOCTYPE (document type declaration) the browser switches from standards mode to quirks mode.

In quirks mode, also known as compatibility mode, the browser simulates an old browser so it can parse old web pages – pages authored before the advent of web standards. A browser in quirks mode pretends to be IE4, IE5 and Navigator 4 so it can render old code as the author intended.

Here's how Wikipedia defines quirks mode:

In computing, quirks mode refers to a technique used by some web
browsers for the sake of maintaining backward compatibility with web
pages designed for older browsers, instead of strictly complying with
W3C and IETF standards in standards mode.

Here's MDN's take:

In the old days of the web, pages were typically written in two
versions: One for Netscape Navigator, and one for Microsoft Internet
Explorer. When the web standards were made at W3C, browsers could not
just start using them, as doing so would break most existing sites on
the web. Browsers therefore introduced two modes to treat new
standards compliant sites differently from old legacy sites.

Now, here's the specific reason why the height: 100% in your code works in quirks mode but not in standards mode:

In standards mode, if the parent element has a height: auto (no height defined), then the percentage heights of child elements will also be treated as height: auto (as per the spec).

This is why the answer to your first question is html { height: 100%; }.

For height: 100% to work in your div, you must set a height on parent elements (more details).

In quirks mode, however, if the parent element has a height: auto, then the percentage heights of child elements are measured relative to the viewport.

Here are three references covering this behavior:

https://www.cs.tut.fi/~jkorpela/quirks-mode.html
https://stackoverflow.com/a/1966377/3597276
https://developer.mozilla.org/en-US/docs/Mozilla_Quirks_Mode_Behavior

TL;DR

Here's what developers need to know in a nutshell:

A browser in quirks mode will render web pages in a way that is
unpredictable, unreliable and often undesirable. So always include a
DOCTYPE for the document to render in standards mode.

Selecting the right DOCTYPE used to be an ambiguous and somewhat
confusing process with many DOCTYPE versions to choose from. But
today the process is as simple as ever. Just use:
<!DOCTYPE html>

Relative dimentions in different doc types

As darksky mentioned, your div has to be contained in another one for the % to take effect. HTML is pretty much just a bunch of containers stacked inside each other. First we have the container, then the container inside of that, and finally the container. Ifg you tell your container to be 100%, it doesn't know a 100% of what, so the right way is something in the lines of:

html,
body {
   margin:0;
   padding:0;
   height:100%;
}
#container {
   min-height:100%;
}

Why does <!DOCTYPE html> break height: 100%?

When you use of <!DOCTYPE html>,you are in standard mode and html and body have height equal his inside content,so you must use this code:

html, body, form, #a {
   height:100%;
}

but when you don't use of DOCTYPE you are in quirks mode and , html and body have default height equal 100%,only use this code:

#a {
   height:100%;
}

position: sticky' not working when 'height' is defined

The issue here is with height, but not the height you thought about. Let's first start by the definition of the sticky position:

A stickily positioned element is an element whose computed position
value is sticky. It's treated as relatively positioned until its
containing block crosses a specified threshold (such as setting top to
value other than auto) within its flow root (or the container it
scrolls within), at which point it is treated as "stuck" until meeting
the opposite edge of its containing block.

The important part here is the last sentence which explain that the position sticky will end when the element reach the edge of its containing block and in your case the containing block of the sticky element is the body and you set the body to be height:100% and you are having an overflow of content.

So when setting the height of main to be 92% and the footer to be 8%, you already made the footer at the oppsite edge of its containing block. Here is an illustration where I added a background color to the body so you can clearly see this:

html,body {  height: 100%;  margin: 0;}html {  background:white;}body {  background:blue;}
#main {  height: 92%;}#landing {  display: flex;  align-items: center;  justify-content: center;  height: 100%;  text-align: center;}#landingContent {  width: 20vw;}#footerNav {  height: 8%;  display: flex;  align-items: center;  position: sticky;  top: 0px;  background:red;  color:#fff;}

<div id="main">    <div id="landing">        <div id="landingContent">            <h1 class="logo">Logo</h1>            <p id="landingParagraph">Lorem ipsum, paragraph content, etc etc.</p>            <button>Button</button>        </div>    </div></div><div id="footerNav">    <div id="footerNavContent">        <h1 class="logo">Logo</h1>    </div></div><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p><p>Hello</p>

Why Certain Doctype Declarations Cause 100%-Height Tables and Divs to Stop Working