How Does Asynchronous JavaScript Execution Happen? and When Not to Use Return Statement

How does Asynchronous Javascript Execution happen? and when not to use return statement?

First of all, passing a function as a parameter is telling the function that you're calling that you would like it to call this function some time in the future. When exactly in the future it will get called depends upon the nature of what the function is doing.

If the function is doing some networking and the function is configured to be non-blocking or asychronous, then the function will execute, the networking operation will be started and the function you called will return right away and the rest of your inline javascript code after that function will execute. If you return a value from that function, it will return right away, long before the function you passed as a parameter has been called (the networking operation has not yet completed).

Meanwhile, the networking operation is going in the background. It's sending the request, listening for the response, then gathering the response. When the networking request has completed and the response has been collected, THEN and only then does the original function you called call the function you passed as a parameter. This may be only a few milliseconds later or it may be as long as minutes later - depending upon how long the networking operation took to complete.

What's important to understand is that in your example, the db.get() function call has long since completed and the code sequentially after it has also executed. What has not completed is the internal anonymous function that you passed as a parameter to that function. That's being held in a javascript function closure until later when the networking function finishes.

It's my opinion that one thing that confuses a lot of people is that the anonymous function is declared inside of your call to db.get and appears to be part of that and appears that when db.get() is done, this would be done too, but that is not the case. Perhaps that would look less like that if it was represented this way:

function getCompletionfunction(result) {
    // do something with the result of db.get
}

// asynchronous Javascript 
db.get('select * from table1', getCompletionFunction);

Then, maybe it would be more obvious that the db.get will return immediately and the getCompletionFunction will get called some time in the future. I'm not suggesting you write it this way, but just showing this form as a means of illustrating what is really happening.

Here's a sequence worth understanding:

console.log("a");
db.get('select * from table1', function(result){
    console.log("b");
});
console.log("c");

What you would see in the debugger console is this:

a
c
b

"a" happens first. Then, db.get() starts its operation and then immediately returns. Thus, "c" happens next. Then, when the db.get() operation actually completes some time in the future, "b" happens.

For some reading on how async handling works in a browser, see How does JavaScript handle AJAX responses in the background?

Asynchronous code, how does it work? Promises and callbacks

I think you have it right.

The function doSth is synchronous and will block the main thread until it's finished. You only provide callback API, but that does not mean it will magically become asynchronous.

Truth is, all JS code you write is synchronous unless you use core JS functions that are defined as asynchronous (e.g., fs.readFile, setTimeout, $.ajax({...}).done). Without them, you can't create asynchronous behavior, you would have to write your own JS core from scratch, with event loop (in fact, I encourage you to google and study what the javascript event loop is, I believe it will clarify many things for you and will give you a much better idea of what is happening in the core). All 3rd party libs out there achieve asynchronous behavior simply because they use these core functions and wrap them with their own code, providing more elegant and higher-level APIs.

The same applies for promises. Yes they are asynchronous, BUT ONLY if you fill them with asynchronous code. Indeed, they have some extra overhead and do not run the code immediately, but if a single promise contains only synchronous code then its eventual execution will block the main thread until it's finished.

How do I return the response from an asynchronous call?

→ For a more general explanation of asynchronous behaviour with different examples, see Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
→ If you already understand the problem, skip to the possible solutions below.

The problem

The A in Ajax stands for asynchronous. That means sending the request (or rather receiving the response) is taken out of the normal execution flow. In your example, $.ajax returns immediately and the next statement, return result;, is executed before the function you passed as success callback was even called.

Here is an analogy which hopefully makes the difference between synchronous and asynchronous flow clearer:

Synchronous

Imagine you make a phone call to a friend and ask him to look something up for you. Although it might take a while, you wait on the phone and stare into space, until your friend gives you the answer that you needed.

The same is happening when you make a function call containing "normal" code:

function findItem() {
    var item;
    while(item_not_found) {
        // search
    }
    return item;
}

var item = findItem();

// Do something with item
doSomethingElse();

Even though findItem might take a long time to execute, any code coming after var item = findItem(); has to wait until the function returns the result.

Asynchronous

You call your friend again for the same reason. But this time you tell him that you are in a hurry and he should call you back on your mobile phone. You hang up, leave the house, and do whatever you planned to do. Once your friend calls you back, you are dealing with the information he gave to you.

That's exactly what's happening when you do an Ajax request.

findItem(function(item) {
    // Do something with the item
});
doSomethingElse();

Instead of waiting for the response, the execution continues immediately and the statement after the Ajax call is executed. To get the response eventually, you provide a function to be called once the response was received, a callback (notice something? call back ?). Any statement coming after that call is executed before the callback is called.

Solution(s)

Embrace the asynchronous nature of JavaScript! While certain asynchronous operations provide synchronous counterparts (so does "Ajax"), it's generally discouraged to use them, especially in a browser context.

Why is it bad do you ask?

JavaScript runs in the UI thread of the browser and any long-running process will lock the UI, making it unresponsive. Additionally, there is an upper limit on the execution time for JavaScript and the browser will ask the user whether to continue the execution or not.

All of this results in a really bad user experience. The user won't be able to tell whether everything is working fine or not. Furthermore, the effect will be worse for users with a slow connection.

In the following we will look at three different solutions that are all building on top of each other:

Promises with async/await (ES2017+, available in older browsers if you use a transpiler or regenerator)
Callbacks (popular in node)
Promises with then() (ES2015+, available in older browsers if you use one of the many promise libraries)

All three are available in current browsers, and node 7+.

ES2017+: Promises with `async/await`

The ECMAScript version released in 2017 introduced syntax-level support for asynchronous functions. With the help of async and await, you can write asynchronous in a "synchronous style". The code is still asynchronous, but it's easier to read/understand.

async/await builds on top of promises: an async function always returns a promise. await "unwraps" a promise and either result in the value the promise was resolved with or throws an error if the promise was rejected.

Important: You can only use await inside an async function or in a JavaScript module. Top-level await is not supported outside of modules, so you might have to make an async IIFE (Immediately Invoked Function Expression) to start an async context if not using a module.

You can read more about async and await on MDN.

Here is an example that elaborates the delay function findItem() above:

// Using 'superagent' which will return a promise.
var superagent = require('superagent')

// This is isn't declared as `async` because it already returns a promise
function delay() {
  // `delay` returns a promise
  return new Promise(function(resolve, reject) {
    // Only `delay` is able to resolve or reject the promise
    setTimeout(function() {
      resolve(42); // After 3 seconds, resolve the promise with value 42
    }, 3000);
  });
}

async function getAllBooks() {
  try {
    // GET a list of book IDs of the current user
    var bookIDs = await superagent.get('/user/books');
    // wait for 3 seconds (just for the sake of this example)
    await delay();
    // GET information about each book
    return superagent.get('/books/ids='+JSON.stringify(bookIDs));
  } catch(error) {
    // If any of the awaited promises was rejected, this catch block
    // would catch the rejection reason
    return null;
  }
}

// Start an IIFE to use `await` at the top level
(async function(){
  let books = await getAllBooks();
  console.log(books);
})();

Current browser and node versions support async/await. You can also support older environments by transforming your code to ES5 with the help of regenerator (or tools that use regenerator, such as Babel).

Let functions accept callbacks

A callback is when function 1 is passed to function 2. Function 2 can call function 1 whenever it is ready. In the context of an asynchronous process, the callback will be called whenever the asynchronous process is done. Usually, the result is passed to the callback.

In the example of the question, you can make foo accept a callback and use it as success callback. So this

var result = foo();
// Code that depends on 'result'

becomes

foo(function(result) {
    // Code that depends on 'result'
});

Here we defined the function "inline" but you can pass any function reference:

function myCallback(result) {
    // Code that depends on 'result'
}

foo(myCallback);

foo itself is defined as follows:

function foo(callback) {
    $.ajax({
        // ...
        success: callback
    });
}

callback will refer to the function we pass to foo when we call it and we pass it on to success. I.e. once the Ajax request is successful, $.ajax will call callback and pass the response to the callback (which can be referred to with result, since this is how we defined the callback).

You can also process the response before passing it to the callback:

function foo(callback) {
    $.ajax({
        // ...
        success: function(response) {
            // For example, filter the response
            callback(filtered_response);
        }
    });
}

It's easier to write code using callbacks than it may seem. After all, JavaScript in the browser is heavily event-driven (DOM events). Receiving the Ajax response is nothing else but an event.
Difficulties could arise when you have to work with third-party code, but most problems can be solved by just thinking through the application flow.

ES2015+: Promises with then()

The Promise API is a new feature of ECMAScript 6 (ES2015), but it has good browser support already. There are also many libraries which implement the standard Promises API and provide additional methods to ease the use and composition of asynchronous functions (e.g., bluebird).

Promises are containers for future values. When the promise receives the value (it is resolved) or when it is canceled (rejected), it notifies all of its "listeners" who want to access this value.

The advantage over plain callbacks is that they allow you to decouple your code and they are easier to compose.

Here is an example of using a promise:

function delay() {
  // `delay` returns a promise
  return new Promise(function(resolve, reject) {
    // Only `delay` is able to resolve or reject the promise
    setTimeout(function() {
      resolve(42); // After 3 seconds, resolve the promise with value 42
    }, 3000);
  });
}

delay()
  .then(function(v) { // `delay` returns a promise
    console.log(v); // Log the value once it is resolved
  })
  .catch(function(v) {
    // Or do something else if it is rejected
    // (it would not happen in this example, since `reject` is not called).
  });

.as-console-wrapper { max-height: 100% !important; top: 0; }

Why do I need to await an async function when it is not supposedly returning a Promise?

All async functions return a promise. All of them.

That promise will eventually resolve with whatever value you return from the async function.

await only blocks execution internal to the async function. It does not block anything outside of the function. Conceptually, an async function starts to execute and as soon as it hits an await instruction, it immediately returns an unfulfilled promise (in the pending state) from the function and the outside execution world gets that promise and continues to execute.

Sometime later, the internal promise that was being awaited will resolve and then the execution of the rest of the internals of the function will continue. Eventually the internals of the function will finish and return a value. That will trigger resolving the promise that was returned from the function with that return value.

FYI, there's a lot of superfluous stuff in your load() function. You can change it from this:

async function load() {
  const data = await new Promise(resolve => {
    setTimeout(() => resolve([1, 2, 3]), 10);
  }).then(data => data.map(i => i * 10));
  console.log(`Data inside the function: ${JSON.stringify(data)}`);
  return data;
}

to this:

function load() {
    return new Promise(resolve => {
        setTimeout(() => resolve([1, 2, 3]), 10);
    }).then(data => data.map(i => i * 10));
}

Then, use it like this:

load().then(result => {
    console.log(result);
});

Or, I prefer to encapsulate the manual creation of promise in their own function like this:

function delay(t, v) {
    return new Promise(resolve => {
        setTimeout(resolve.bind(null, v), t);
    });
}

function load() {
    return delay(10, [1, 2, 3]).then(data => data.map(i => i * 10));
}

And, it turns out this little delay() function is generally useful in lots of places where you want to delay a promise chain.

Thanks to everyone participating and providing me with insight. But I'm still confused how should I be using await and async.

First off, most of the time you only mark a function async if you need to use await inside the function.

Second, you most commonly use await (from within an async function) when you have multiple asynchronous operations and you want to sequence them - often because the first one provides a result that is used as input to the second. You can use await when all you have is a single asynchronous operation, but it doesn't really offer much of an advantage over a simple .then().

Here are a few examples of good reasons to use async/await:

Sequencing multiple asynchronous operations

Imagine you have getFromDatabase(), getTheUrl() and getTheContent() that are all asynchronous. If any fails, you would want to just reject the returned promise with the first error.

Here's how this looks without async/await:

function run() {
    return getFromDatabase(someArg).then(key => {
        return getTheURL(key);
    }).then(url => {
        return getTheContent(url);
    }).then(content => {
         // some final processing
         return finalValue;
    });
}

Here's how this looks with async/await:

async function run(someArg) {
    let key = await getFromDatabase(someArg);
    let url = await getTheURL(key);
    let content = await getTheContent(url);
    // some final processing
    return finalValue;        
}

In both cases, the function returns a promise that resolves with the finalValue so these two implementations are used the same by the caller:

run(someArg).then(finalValue => {
    console.log(finalValue);
}).catch(err => {
    console.log(err);
});

But, you will notice that the async/await implementation has more of a serialized, synchronous look to it and looks more like non-asynchronous code. Many find this easier to write, easier to read and easier to maintain. The more processing you have between steps, including branching, the more advantages accrue to the async/await version.

Automatically catching both rejected promises and synchronous exceptions

As I said earlier, async functions always return a promise. They also have to built-in error handling that automatically propagates errors back to that returned promise.

It goes without saying that if you manually return a promise from the async function and that promise rejects, then the promise returned from the async function will reject.

But also, if you are using await and any promise you are awaiting rejects and you don't have a .catch() on the promise and don't have a try/catch around it, then the promise the function returns will automatically reject. So, back in our previous example of this:

async function run(someArg) {
    let key = await getFromDatabase(someArg);
    let url = await getTheURL(key);
    let content = await getTheContent(url);
    // some final processing
    return finalValue;        
}

If any of the three promises that are being awaited reject, then the function will short circuit (stop executing any more code in the function) and reject the async returned promise. So, you get this form of error handling for free.

Then lastly, an async function also catches synchronous exceptions for you and turns them into a rejected promise.

In a normal function that returns a promise such as we had earlier:

function run() {
    return getFromDatabase(someArg).then(key => {
        return getTheURL(key);
    }).then(url => {
        return getTheContent(url);
    }).then(content => {
         // some final processing
         return finalValue;
    });
}

If getFromDatabase() throws a synchronous exception (perhaps triggered because someArg is invalid), then this overall function run() will throw synchronously. That means that for the caller to catch all possible errors from run(), they have to both surround it with a try/catch to catch the synchronous exceptions and use a .catch() to catch the rejected promise:

try {
    run(someArg).then(finalValue => {
        console.log(finalValue);
    }).catch(err => {
        console.log(err);
    });
} catch(e) {
    console.log(err);
}

This is messy and a bit repetitive. But, when run() is declared async, then it will NEVER throw synchronously because any synchronous exception is automatically converted to a rejected promise so you can be sure you are capturing all possible errors when it's written this way:

async function run(someArg) {
    let key = await getFromDatabase(someArg);
    let url = await getTheURL(key);
    let content = await getTheContent(url);
    // some final processing
    return finalValue;        
}

// will catch all possible errors from run()
run(someArg).then(finalValue => {
    console.log(finalValue);
}).catch(err => {
    console.log(err);
});

Should I always call all of my function with an await?

First, you would only ever use await with a function that returns a promise as await offers no usefulness if the function does not return a promise (just adding to the clutter of your code if not needed).

Second, whether you use await or not depends upon the context of both the calling function (since you HAVE to be in an async function to use await and on the flow of logic and whether it benefits from using await or not.

Places where it's pointless to use await

async function getKey(someArg) {
    let key = await getFromDatabase(someArg);
    return key;
}

The await here isn't doing anything useful. You're not sequencing multiple async operations and you're not doing any processing on the return value. You can accomplish the exact same code by just returning the promise directly:

async function getKey(someArg) {
    return getFromDatabase(someArg);
}

And, if you know that getFromDatabase() never throws synchronously, you can even remove the async from the declaration:

function getKey(someArg) {
    return getFromDatabase(someArg);
}

Let's say I'm writing a code composed of multiple functions within multiple files. If I end up using a library which returns a Promise or it's an async function, should I trace back all my function calls from the asynchronous point to the entry point of the application and add an await before all the function calls after making them async?

This is a bit too general of an ask that's hard to answer in a general case. Here are some thoughts along this general direction:

Once any part of your result that you're trying to return from your function A() is asynchronous or uses any asynchronous operation to obtain, the function itself is asynchronous. In plain Javascript, you can never return an asynchronous result synchronously so your function must use an asynchronous method to return the result (promise, callback, event, etc...).
Any function B() that calls your asynchronous function A() that is also trying to return a result based on what it gets from A() is now also asynchronous and also must communicate its result back using an asynchronous mechanism. This is true for a function C() that calls B() and needs to communicate back its result to the caller. So, you can say that asynchronous behavior is infectious. Until you reach some point in the call chain where you no longer need to communicate back a result, everything has to use asynchronous mechanisms to communicate the result, error and completion.
There's no specific need to mark a function async unless you specifically need one of the benefits of an async function such as the ability to use await inside that function or the automatic error handling it provides. You can write functions that returning promises just fine without using async on the function declaration. So, "NO" I don't go back up the call chain making everything async. I only make a function async if there's a specific reason to do so. Usually that reason is that I want to use await inside the function, but there is also the automatic catching of synchronous exceptions that get turned into promise rejections that I described earlier. You would not generally need that with well behaved code, but it is sometimes useful with poorly behaved code orcode with an undefined behavior.
await is also only used when there's a specific reason for it. I don't just automatically use it on every function that returns a promise. I've described above reasons to use it. One can still use .then() just fine for processing the result from a single function call that returns a promise. In some cases, it's just a matter of personal style whether you want to use .then() or await and there is no particular reason it has to be one one way or the other.

Or maybe I should just get into the habit of calling all my functions with an await regardless of whether they are async or not?

Absolutely NOT! First off, the last thing you want to do is to take perfectly synchronous code and unnecessarily make it asynchronous or even make it look asynchronous. Asynchronous code (even with async and await) is more complicated to write, debug, understand and maintain than synchronous code so you would never want to unnecessarily make synchronous code into asynchronous code by adding async/await into it:

For example, you would never do this:

async function random(min, max) {
    let r = await Math.random();
    return Math.floor((r * (max - min)) + min);
}

First off, this is a perfectly synchronous operation that can be coded like this:

function random(min, max) {
    let r = Math.random();
    return Math.floor((r * (max - min)) + min);
}

Second off, that first async implementation has made the function a lot hard to use as it now has an asynchronous result:

 random(1,10).then(r => {
     console.log(r);
 });

Instead of just the simple synchronous use:

 console.log(random(1,10));

When does the useEffect's callback's return statement execute?

Your understanding of the sequence of events is correct. The only thing missing is the precise timing of the effect callbacks and cleanup.

When the component re-renders, any useEffects will have their dependency arrays analyzed for changes. If there has been a change, then that effect callback will run. These callbacks are guaranteed to run in the order that they're declared in the component. For example, below, a will always be logged just before b.

const App = () => {
    const [num, setNum] = React.useState(0);
    React.useEffect(() => {
      setInterval(() => {
        setNum(num => num + 1);
      }, 1000);
    }, []);
    React.useEffect(() => {
      console.log('a', num);
    }, [num]);
    React.useEffect(() => {
      console.log('b', num);
    }, [num]);
    return num;
}

ReactDOM.render(<App />, document.querySelector('.react'));

<script crossorigin src="https://unpkg.com/react@16/umd/react.development.js"></script>
<script crossorigin src="https://unpkg.com/react-dom@16/umd/react-dom.development.js"></script>
<div class='react'></div>

How Does Asynchronous JavaScript Execution Happen? and When Not to Use Return Statement