How to Keep Parents of Floated Elements from Collapsing

Floating elements causes the parent to collapse. HTML / CSS

Since you needed the reasons, I think this post explains it very well. Apart from reasons, it also has some solutions to tackle it.

About float:

 when an element is floated it is taken out of the normal flow of the document. 
 It is shifted to the left or right until it touches the edge of it's containing
 box or another floated element.

Reference: https://developer.mozilla.org/en-US/docs/Web/CSS/float

About clear:

The clear CSS property specifies whether an element can be next to floating 
elements that precede it or must be moved down (cleared) below them.

The clear property applies to both floating and non-floating elements.
When applied to non-floating blocks, it moves the border edge of the 
element down until it is below the margin edge of all relevant floats.
This movement (when it happens) causes margin collapsing not to occur.

When applied to floating elements, it moves the margin edge of the element
below the margin edge of all relevant floats. This affects the position of later
floats, since later floats cannot be positioned higher than earlier ones.

The floats that are relevant to be cleared are the earlier floats within the 
same block formatting context.

Reference: https://developer.mozilla.org/en-US/docs/Web/CSS/clear

How to stop parents of absolutely positioned elements from collapsing

You can't : once the child is in absolute position, it's virtually 'outside' of the parent (in appearance).

what you can do, if you have included jquery, is use this unelegant hack :

$(".absolutepos").each(function() {
    $(this).parent().css("height",$(this).height());
});

and add the "absolutepos" class when placing the div in absolute position :

<div id="outer" style="position: relative;">
    <div id="inner absolutepos" style="position: absolute; height: 100px;">
        <p>This is the inner content.</p>
    </div>            
</div>

How to stop parents of absolutely positioned elements from collapsing

You can't : once the child is in absolute position, it's virtually 'outside' of the parent (in appearance).

what you can do, if you have included jquery, is use this unelegant hack :

$(".absolutepos").each(function() {
    $(this).parent().css("height",$(this).height());
});

and add the "absolutepos" class when placing the div in absolute position :

<div id="outer" style="position: relative;">
    <div id="inner absolutepos" style="position: absolute; height: 100px;">
        <p>This is the inner content.</p>
    </div>            
</div>

Why do non-floating parents of floating elements collapse?

Take a look at this section in the w3c CSS 2.1 spec: 9.5 Floats

Note this image at the bottom, just before section 3.5.1:

... this seems to provide the use-case I'm after:

Case description: You have an image floated inside a paragraph with a great enough height that it extends well below its sibling text. In general, you'd want the text in the subsequent paragraph to wrap around this image as well. The way to achieve this is to allow the image to protrude outside the containing paragraph. Otherwise -- if the first, containing paragraph's height extends all the way down to wrap the image -- the subsequent paragraph gets pushed down completely below the image, leaving a large white-space between the texts of the two paragraphs.

However: more often than not we don't want this effect when using floats. So often we need floats to layout main areas of a web page (lest we resort to tables), and in these cases we typically need a container to expand to include whatever is inside in its height calculation.

My wish: It seems, then, that there should be a CSS property along the lines of:

box-model-height: [ include-floats | exclude-floats ];

Browsers could have the default on paragraphs as "exclude-floats", and all other blocks as "include-floats".

Or if that would break too many designs, "exclude-floats" could be the default everywhere, and we could still fix the situation, wherever we need to, entirely in the style-sheet, instead of requiring a class-name (e.g. clearfix) in the markup.

Margin collapsing with floated element, why there is an extra margin added?

Before I start, the issue of scrollbars being rendered in all browsers but Firefox is a separate issue from what is being asked about here. Firefox does not collapse margins between a parent element and its children when the parent's min-height results in the margins not being adjoining. It's also a known spec violation in Firefox that's being worked on and yet to be fixed.

Now, on to the issue at hand. From section 9.5.1 (on floats):

4. A floating box's outer top may not be higher than the top of its containing block. When the float occurs between two collapsing margins, the float is positioned as if it had an otherwise empty anonymous block parent taking part in the flow. The position of such a parent is defined by the rules in the section on margin collapsing.

The last sentence in this quote is awkward, but "the rules" refer (and link) to the definition of collapsing through. While the specific text that you cite from that section is relevant, it doesn't explain why the floats respect the margin of the in-flow div.

This does:

If the top and bottom margins of a box are adjoining, then it is possible for margins to collapse through it. In this case, the position of the element depends on its relationship with the other elements whose margins are being collapsed.

• [...]

• Otherwise, either the element's parent is not taking part in the margin collapsing, or only the parent's bottom margin is involved. The position of the element's top border edge is the same as it would have been if the element had a non-zero bottom border.

Note the last sentence. Having a non-zero bottom border cancels margin collapsing, as you know. This means that the floats are positioned as if the bottom margins of the in-flow div and the body element did not collapse, resulting in the floats appearing to respect the bottom margin of the in-flow div.

How do I tell that the floats specifically respect the bottom margin of the in-flow div and not the collapsed margin? By giving body a larger bottom margin than that of the in-flow div and observing that it does not affect the position of the floats:

html {
  background: red;
}

body {
  margin: 0;
  margin-bottom: 100px;
  min-height: 100vh;
  background-color: green;
}

div {
  min-height: 50px;
  background-color: pink;
  margin-bottom: 50px;
}
.l {
  width:45%;
  height:50px;
  float:left;
  margin:0;
}
.r {
  width:45%;
  height:50px;
  float:right;
  margin:0;
}

<div></div>
<div class="l"></div>
<div class="r"></div>

#mainContainer{
    width: 1000px;
    /*height: 1000px;*/
    height:auto;
    margin-left:auto;
    margin-right:auto;
    background-color: #ff6600;
    padding-bottom: 20px;

    overflow: hidden; /* <--- here */
}

.main {  border-style: solid;  border-color: yellow; /* overflow: auto;*/}
.first {  width: 200px;  height: 100px;  float: left;  border: 10px solid green;}
.second {  width: 200px;  /*height: 50px;*/  border: 10px solid blue;}

<div class="main">  <div class="first">test1</div>  <div class="second">test2</div></div>