How to Use .Notparallel in Makefile Only on Specific Targets

How can I use .NOTPARALLEL in makefile only on specific targets?

Create a target specifying the four targets that can be executed in parallel & include this and last_label in the all target:

intermediate: label1 label2 label3 label4

all:
        $(MAKE) intermediate
        $(MAKE) last_label

This would execute the targets specified within intermediate in parallel, but intermediate and last_label would be forced consecutively.

(Note that the leading space before $(MAKE) is a TAB character.)

How can I use .NOTPARALLEL in makefile only on targets from list?

One possible approach is to use a synchronization on a file level within the recipe. It's not of course the best (as it will still consume make's job), but should not overload the system by running n multiple-threaded programs.

Sample Makefile:

$ cat Makefile
all: $(foreach number,1 2 3 4 5 6 7 8 9 10,$(addsuffix /$(number),slow_target fast_target))

slow_target/%:
        @echo $@
        @python3 -c 'import time; time.sleep(5)'

fast_target/%:
        @( \
          flock 9 || exit 1; \
          echo $@; \
          python3 -c 'import time; time.sleep(5)' \
        ) 9> fast_target.lock

Output:

$ make -j8 | ts
May 19 23:45:06 slow_target/1
May 19 23:45:06 slow_target/2
May 19 23:45:06 fast_target/1
May 19 23:45:06 slow_target/3
May 19 23:45:06 slow_target/4
May 19 23:45:11 fast_target/2
May 19 23:45:11 slow_target/5
May 19 23:45:11 slow_target/6
May 19 23:45:11 slow_target/7
May 19 23:45:16 fast_target/3
May 19 23:45:16 slow_target/8
May 19 23:45:16 slow_target/9
May 19 23:45:21 fast_target/4
May 19 23:45:21 slow_target/10
May 19 23:45:26 fast_target/5
May 19 23:45:31 fast_target/6
May 19 23:45:36 fast_target/7
May 19 23:45:41 fast_target/8
May 19 23:45:46 fast_target/9
May 19 23:45:51 fast_target/10

Note that slow_targets are started in parallel, but only one fast_target at a time is started.

EDIT

flock acquires a lock on a given file or a given file descriptor. The lock by default is exclusive, meaning that only one flock call can continue and all others will block until the lock is released. Therefore if multiple targets are invoked in parallel, only one flock will proceed, while others will wait till the lock is released.

The statement in the recipe is based on flock's manual. It opens up file descriptor 9 on a fast_target.lock file for a given block. At the beginning flock acquires an exclusive lock on this descriptor (thus fast_target.lock file), which will automatically be released when the descriptor is closed (that is when the block with redirection ends). When that happens, one of other flocks will proceed, effectively allowing only one fast_target recipe execution.

EDIT 2

If all fast_targets are known and always executed, they can be dynamically scheduled in a sequence, like so:

$ cat Makefile2
SLOW_TARGETS := $(addprefix slow_target/,1 2 3 4 5 6 7 8 9 10)
FAST_TARGETS := $(addprefix fast_target/,1 2 3 4 5 6 7 8 9 10)

all: $(SLOW_TARGETS) $(FAST_TARGETS)

slow_target/%:
        @echo $@
        @python3 -c 'import time; time.sleep(5)'

fast_target/%:
        @echo $@: $^
        @python3 -c 'import time; time.sleep(5)'

$(foreach target,$(FAST_TARGETS), \
  $(eval evaluated_targets += $(target)) \
  $(eval next_target := $(word 2,$(wordlist $(words $(evaluated_targets)),$(words $(FAST_TARGETS)),$(FAST_TARGETS)))) \
  $(eval $(if $(next_target),$(next_target): $(target))) \
)

This will iterate the list of known fast_targets and for every target it will define a dependency to the previous one. This will result in a strict sequence to run, so none of them will be run in parallel. Additionally it will not block make jobs, so the executors will be available for other targets instead of being blocked on the flock.

$ make -f Makefile2 -j8 | ts
May 20 22:38:35 slow_target/1
May 20 22:38:35 slow_target/2
May 20 22:38:35 slow_target/3
May 20 22:38:35 slow_target/4
May 20 22:38:35 slow_target/5
May 20 22:38:35 slow_target/6
May 20 22:38:35 slow_target/7
May 20 22:38:35 slow_target/8
May 20 22:38:40 slow_target/9
May 20 22:38:40 slow_target/10
May 20 22:38:40 fast_target/1:
May 20 22:38:45 fast_target/2: fast_target/1
May 20 22:38:51 fast_target/3: fast_target/2
May 20 22:38:56 fast_target/4: fast_target/3
May 20 22:39:01 fast_target/5: fast_target/4
May 20 22:39:06 fast_target/6: fast_target/5
May 20 22:39:11 fast_target/7: fast_target/6
May 20 22:39:16 fast_target/8: fast_target/7
May 20 22:39:21 fast_target/9: fast_target/8
May 20 22:39:26 fast_target/10: fast_target/9

Note that due to dependency, it is not possible to run e.g. only fast_target/10 without further logic manipulation, as all previous targets will also be run. This would not be the case with the flock approach.

How can I use Target-specific Variable Values and call other rules whilst using parallel execution?

Why not just pass through the values as well?

debug: CFLAGS += $(DEBUGFLAGS)
debug:
        $(MAKE) clean CFLAGS='$(CFLAGS)'
        $(MAKE) all CFLAGS='$(CFLAGS)'

make: disable parallel building in subdirectory for single target only

If I understood you correctly, then when you run the recursive make that builds the check target you can just pass -j1 specifically, to ensure that it runs serially:

check: ; $(MAKE) -j1 ...

Execute a Makefile in parallel except for some rules

This is really not right:

MyObjects: $(OBJ)
        $(MAKE) -j $(OBJ)

This means that before make tries to build the MyObjects target, it will first try to update all the $(OBJ) files. Once that's all done, then it will try to build the MyObjects target by recursively invoking make to rebuild them again. Obviously that's not what you want. Plus you're using -j which is basically "infinitely parallel" and is likely (if you have enough object files) to bring your system to its knees.

You want something like this:

MyObjects:
        $(MAKE) -j5 $(OBJ)

As for your second question about trying to rebuild targets, there's no way we can help without some kind of specific example. Typically this happens because your rules are written incorrectly, and they don't actually update the target you told make they would. So for example, you have a target recipe_a but the rule for recipe_a updates some other target, not recipe_a.

I'll add a few notes based on your second question. Probably if you don't get it after this you should take this off of StackOverflow and ask on the help-make@gnu.org mailing list, or else consider breaking this up and asking several specific StackOverflow questions.

First, why you see make[1]: '15.o' is up to date. for every file in your recursive make: because make always prints that message for every target on the command line, so if you run make 1.o 2.o 3.o ... (doesn't matter whether you use -j or not or what value of -j you use) you'll get that message for every target which doesn't need to be rebuilt. Just as if you ran that same make command from the command line yourself.

Second, why you don't get a.out is up to date, because a.out is NOT up to date. It depends on the build target, and the file build doesn't exist, and thus it's out of date, and so it must be rebuilt every time. And that means anything that depends on the build target, like a.out, must be rebuilt every time. Which explains why it always re-runs the zip command.

Third, the behavior with all.c is because if you create a pattern rule like %.c: with no prerequisites, that tells make that it can create ANY file with a .c extension by running that command. Well, one of the targets you asked make to build is the all target. Since you didn't declare that as a .PHONY target, make tries to build it. Normally that attempt fails because make can't find any rules that know how to build all so nothing happens, but after you tell make how to build a .c file out of nothing (no prerequisites), then when make wants to build all it looks in its internal database of predefined rules and sees a pattern rule % : %.c, which tells make how to build an executable from a source file with the same name (on UNIX systems executables don't have any suffix like .exe: they're just make or cc etc.) So, make tries to run those rules and they fail.

For any target which you don't expect to actually be created, like all, clean, etc. you should declare them to be .PHONY so make won't try to build them.

As for your problem. I think the simplest thing to do is push the entire build of the zip file down into the recursive make, rather than trying to build the objects only in the recursive make. Something like this:

NAMES = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

SRC = $(addsuffix .c,$(NAMES))
OBJ = $(patsubst %.c,%.o,$(SRC))

all: recurse

recurse: non-parallel-targets
        $(MAKE) -j8 a.out PARALLEL=true

ifneq($(PARALLEL),true)
.NOTPARALLEL:
endif

%.o: %.c
        cp $< $@

a.out: $(OBJ)
        zip $@ $(OBJ)

init: $(SRC)

clean: 
        -rm *.o
        -rm *.out

.PHONY: all clean init

How can I force gnu make to not build recipe in parallel?

This is a horrible kludge, but it will do the job:

b.cpp: a.cpp

c.cpp: b.cpp

Or if there are actually a lot of these, you can have a few stiff drinks and do this:

c-sources = $(sources:.xxx=.cpp)

ALLBUTFIRST = $(filter-out $(firstword $(c-sources)), $(c-sources))
ALLBUTLAST = $(filter-out $(lastword $(c-sources)), $(c-sources))
PAIRS = $(join $(ALLBUTLAST),$(addprefix :,$(ALLBUTFIRST)))

$(foreach pair,$(PAIRS),$(eval $(pair)))

(This works in GNUMake, I don't know about other versions.)

Parallel makefile requires dependency ordering

My preference is for

release:
    $(MAKE) clean
    $(MAKE) test1

This forces the two targets to be made consecutively without disturbing their inner parallelism (if any).

---

Related Topics