Swift Package Manager Unable to Compile Ncurses Installed Through Homebrew

Swift package manager unable to compile ncurses installed through Homebrew

Problem

The main problem are conflicts of the header files, since ncurses is also supplied in /Applications/Xcode.app/.../MacOSX10.14.sdk/usr/include.

A common pure C solution in such situations is to just specify the custom include and lib directories with -I repective -L and it would work, see my answer regarding C ncurses here: https://stackoverflow.com/a/56623033/2331445

This approach does not seem to work with the Swift package manager. But that doesn't mean it's not possible with a little effort.

Possible Solution

We need to make sure that the ncurses header files provided by the macOS SDK are ignored. We can do this by specifying the -Xcc -D__NCURSES_H parameter for the swift build command.

This works because in the header file, there is this typical:

#ifndef __NCURSES_H
#define __NCURSES_H
...
#endif

The problem, of course, is that our custom installation of ncurses using Brew is also affected. But we can work around it:

  • copy the new ncurses header files into our Sources/Cncurses directory
  • replace __NCURSES_H through something different, e.g. __CNCURSES_H (note leading 'C')
  • then make sure that all further nested includes are first searched in our local include directory by replace the angle brackets of the includes with quotes, so e.g. instead of #include <ncursesw/unctrl.h> the form '#include "ncursesw/unctrl.h"' is used

This can actually be done with the following command line commands:

cd Sources/Cncurses
cp -r /usr/local/Cellar/ncurses/6.1/include include
find . -name '*.h' -exec sed -i '' 's/__NCURSES_H/__CNCURSES_H/g' {} \;
find . -name '*.h' -exec sed -i '' -E -e "s/<(.*(`find . -name '*.h' -exec basename {} \; | paste -sd "|" -`))>/\"\1\"/g" {} \;

The last statement may require some explanation. With the help of an echo command, you can look at the generated sed expression, i.e. if you execute

echo "s/<(.*(`find . -name '*.h' -exec basename {} \; |  paste -sd "|" -`))>/\"\1\"/g" 

you get the following output:

s/<(.*(termcap.h|form.h|term.h|panel.h|ncurses.h|termcap.h|cursesp.h|cursesf.h|etip.h|form.h|cursesw.h|nc_tparm.h|unctrl.h|cursesapp.h|term.h|cursslk.h|panel.h|ncurses.h|tic.h|eti.h|ncurses_dll.h|term_entry.h|menu.h|cursesm.h|curses.h|curses.h|cncurses.h))>/"\1"/g

As you can see, it searches and replaces only local available include files.

Test

For a test we need a simple ncurses example program. It should be built and we should make sure that the correct version of the library is used.

module.modulemap

My header file is called cncurses.h. The module.modulemap looks like this:

module cncurses [system] 
{
umbrella header "cncurses.h"
link "ncurses"
export *
}

cncurses.h

cncurses.h is a one-liner, it imports our copied and customized ncurses.h file from our local include folder:

#include "include/ncurses.h"

main.swift

In the NcursesExample folder we have main.swift where we have a simple cncurses swift app:

import cncurses

initscr()
curs_set(0)
move(5, 10)
addstr("NCURSES")
move(10, 10)
addstr("Hello World!")
refresh()

select(0, nil, nil, nil, nil)

Package.swift

Please note here the pkgConfig: "ncurses" in the systemLibrary targets:

// swift-tools-version:5.0
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
name: "NcursesExample",
dependencies: [
],
targets: [
.systemLibrary(name: "cncurses", pkgConfig: "ncurses"),
.target(name: "NcursesExample", dependencies: ["cncurses"]),
.testTarget(
name: "NcursesExampleTests",
dependencies: ["NcursesExample"]),

]
)

Build

For pkg-config to do its job properly, we must first call the following:

export PKG_CONFIG_PATH="/usr/local/opt/ncurses/lib/pkgconfig"

Finally we initiate the build with:

swift build -Xcc -D__NCURSES_H 

So first we should test if the correct ncurses lib was used. We can do that with:

otool -L .build/x86_64-apple-macosx/debug/NcursesExample

Among other lines, the output contains this:

/usr/local/opt/ncurses/lib/libncursesw.6.dylib (compatibility version 6.0.0, current version 6.0.0)

which looks promising.

Finally calling the binary:

ncurses demo

Xcode Project

If you want to generate a Xcode project, use the following command:

swift package generate-xcodeproj

Then load the project in Xcode and

  • select the project node
  • in Build settings enter Preprocessor in the search field in the upper right
  • under Apple Clang - Preprocessing / Preprocess Macros add __NCURSES_H=1 for Debug and Release

Clang on macOS fails linking lmenu from ncurses

You can do the following:

brew install ncurses

Since macOS already contains a ncurses version brew installs its alternative version in /usr/local/opt/ncurses.

So that the compiler and linker can access it, your build command should now look like this:

gcc -I/usr/local/opt/ncurses/include -L/usr/local/opt/ncurses/lib libmenutest.c -o test -lmenu -lncurses

When you finally call your program, the following is output:

This worked :^) 

CMake

For the folks using CMake your CMakeLists.txt could look like this:

cmake_minimum_required(VERSION 3.14)
project(libmenutest C)

set(CMAKE_C_STANDARD 99)

include_directories(/usr/local/opt/ncurses/include)

link_directories(/usr/local/opt/ncurses/lib)

add_executable(libmenutest libmenutest.c)

target_link_libraries(libmenutest menu ncurses)

Linux: Compile using a specific ncurses version when both V6 and V5 are installed?

Debian has several packages for ncurses (I find the bug-reporting links easier to navigate than the package-tracking). At the moment, buster is "oldstable". That has development packages for ABI 5 and 6:

  • libncurses-dev
  • libncursesw-dev
  • libncurses5-dev
  • libncursesw5-dev

Each of those has a package-page which includes a list of files for each architecture, e.g., this, and for example

/usr/lib/x86_64-linux-gnu/pkgconfig/form.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/formw.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/menu.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/menuw.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/ncurses++.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/ncurses++w.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/ncurses.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/ncursesw.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/panel.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/panelw.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/tic.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/tinfo.pc

Generally, the filename for the ".pc" files is useful as a parameter to pkg-config, e.g.,

pkg-config --cflags --libs libncurses-dev

which prints the -I, -D, -L and -l options you'd use in a makefile or script.

You don't have to search through the webpages for that;

dpkg -l | grep 'ncurses'

lists the package with "ncurses" in their name, and for the example shown

dpkg -L libncurses-dev | grep -F .pc

shows the pkg-config files.

In a command-line, I could do this:

gcc -c $(pkg-config --cflags libncurses-dev) foo.c

for just the compiler, or to compile/link:

gcc -o foo $(pkg-config --cflags --libs libncurses-dev) foo.c

The package also has the "*-config" scripts which give similar information, without using pkg-config, e.g.,

/usr/bin/ncurses5-config
/usr/bin/ncurses6-config
/usr/bin/ncursesw5-config
/usr/bin/ncursesw6-config

In the Debian development packages, it appears that there are ".pc" files only for ABI 6. But the "-config" files are provided for both ABI 5 and 6. So you should be able to compile with ABI 5, e.g,.

gcc -c $(ncurses5-config --cflags libncurses-dev) foo.c

Use ansible package module to work with apt and homebrew

If you want a single set of tasks that flexible enough for for multiple Linux
package managers and macOS brew, the choice is either more logic or more
duplication.

These three patterns should help. They still have repetition and
boilerplate code, but that's the territory we're in with Ansible plays
for cross-platform.

  1. Declare become: yes (root) globally for Linux only
  2. Address packages that need platform-specific treatment as-needed with when
    • This might be --head for brew, or setting up a PPA for apt, etc
  3. Map package name discrepancies with variables
    • For example: brew install ncurses, apt install libncurses5-dev, and dnf install ncurses-devel are all the same library.

1) Declare become: yes (root) globally for Linux only

For Linux hosts, switching to root for installation is the intended behavior.
For macOS a la Homebrew, installing as root is not good. So, we need become: no (false) when using brew, and become: yes (true) otherwise (for
Linux).

In your example the become directive is nested inside each task ("step"). To
prevent duplication, invoke become at a higher lexical scope, before the tasks
start. The subsequent tasks will then inherit the state of become, which is
set based on a conditional expression.

Unfortunately a variable for become at the root playbook scope will be
undefined and throw an error before the first task is run:

# playbook.yml
- name: Demo
hosts: localhost
connection: local
# This works
become: True
# This doesn't - the variable is undefined
become: "{{ False if ansible_pkg_mgr == 'brew' else True }}"
# Nor does this - also undefined
become: "{{ False if ansible_os_family == 'Darwin' else True }}"

tasks:
# ...

To fix this, we can store the tasks in another file and import them, or
wrap the tasks in a block. Either of these patterns will provide
a chance to declare become with our custom variable value in time for the
tasks to pick it up:

# playbook.yml
---
- name: Demo
hosts: localhost
connection: local
vars:
# This variable gives us a boolean for deciding whether or not to become
# root. It cascades down to any subsequent tasks unless overwritten.
should_be_root: "{{ true if ansible_pkg_mgr != 'brew' else false }}"

# It could also be based on the OS type, but since brew is the main cause
# it's probably better this way.
# should_be_root: "{{ False if ansible_os_family == 'Darwin' else True }}"

tasks:
# Import the tasks from another file, which gives us a chance to pass along
# a `become` context with our variable:
- import_tasks: test_tasks.yml
become: "{{ should_be_root }}"

# Wrapping the tasks in a block will also work:
- block:
- name: ncurses is present
package:
name: [libncurses5-dev, libncursesw5-dev]
state: present
- name: cmatrix is present
package:
name: cmatrix
state: present
become: "{{ should_be_root }}"

Now there is a single logic check for brew and a single before directive
(depending on which task pattern above is used). All tasks will be executed as
the root user, unless the package manager in use is brew.

2) Address packages that need platform-specific treatment as-needed with when

The Package Module is a great convenience but it's quite limited. By
itself it only works for ideal scenarios; meaning, a package that doesn't
require any special treatment or flags from the underlying package manager. All
it can do is pass the literal string of the package to install, the state, and
an optional parameter to force use of a specific package manager executable.

Here's an example that installs wget with a nice short task and only becomes
verbose to handle ffmpeg's special case when installed with brew:

# playbook.yml
# ...
tasks:
# wget is the same among package managers, nothing to see here
- name: wget is present
when: ansible_pkg_mgr != 'brew'
package:
name: wget
state: present

# This will only run on hosts that do not use `brew`, like linux
- name: ffmpeg is present
when: ansible_pkg_mgr != 'brew'
package:
name: ffmpeg
state: present

# This will only run on hosts that use `brew`, i.e. macOS
- name: ffmpeg is present (brew)
when: ansible_pkg_mgr == 'brew'
homebrew:
name: ffmpeg
# head flag
state: head
# --with-chromaprint --with-fdk-aac --with-etc-etc
install_options: with-chromaprint, with-fdk-aac, with-etc-etc

The play above would produce this output for ffmpeg against a Linux box:

TASK [youtube-dl : ffmpeg is present] ******************************************
ok: [localhost]

TASK [youtube-dl : ffmpeg is present (brew)] ***********************************
skipping: [localhost]

3) Map package name discrepancies with variables

This isn't specifically part of your question but it's likely to come up next.

The Package Module docs also mention:

Package names also vary with package manager; this module will not "translate"
them per distro. For example libyaml-dev, libyaml-devel.

So, we're on our own to handle cases where the same software uses different
names between package manager platforms. This is quite common.

There are multiple patterns for this, such as:

  • Use separate variable files for each OS/distro and import them
    conditionally
  • Use a role with its own variables
  • Use the same package manager across platforms, such as Homebrew or
    Conda
  • Compile everything from source via git

None of them are very pleasant. Here is an approach using a role. Roles do
involve more boilerplate and directory juggling, but in exchange they provide
modularity and a local variable environment. When a set of tasks in a role
requires more finagling to get right, it doesn't end up polluting other task
sets.

# playbook.yml
---
- name: Demo
hosts: localhost
connection: local
roles:
- cmatrix

# roles/cmatrix/defaults/main.yml
---
ncurses:
default:
- ncurses
# Important: these keys need to exactly match the name of package managers for
# our logic to hold up
apt:
- libncurses5-dev
- libncursesw5-dev
brew:
- pkg-config
- ncurses

# roles/cmatrix/tasks/main.yml
---
- name: cmatix and its dependencies are present
become: "{{ should_be_root }}"
block:
- name: ncurses is present
package:
name: '{{ item }}'
state: latest
loop: "{{ ncurses[ansible_pkg_mgr] | default(ncurses['default']) }}"

- name: cmatrix is present
when: ansible_pkg_mgr != 'brew'
package:
name: cmatrix
state: present

The task for ncurses looks for an array of items to loop through keyed by the
corresponding package manager. If the package manager being used is not defined
in the variable object, a Jinja default filter is employed to reference the
default value we set.

With this pattern, adding support for another package manager or additional
dependencies simply involves updating the variable object:

# roles/cmatrix/defaults/main.yml
---
ncurses:
default:
- ncurses
apt:
- libncurses5-dev
- libncursesw5-dev
# add a new dependency for Debian
- imaginarycurses-dep
brew:
- pkg-config
- ncurses
# add support for Fedora
dnf:
- ncurses-devel

Combining everything into a real play

Here's a full example covering all three aspects. The playbook has two roles
that each use the correct become value based on a single variable. It also
incorporates an special cases for cmatrix and ffmpeg when installed with
brew, and handles alternate names for ncurses between package managers.

# playbook.yml
---
- name: Demo
hosts: localhost
connection: local
vars:
should_be_root: "{{ true if ansible_pkg_mgr != 'brew' else false }}"
roles:
- cmatrix
- youtube-dl
# roles/cmatrix/defaults/main.yml
ncurses:
default:
- ncurses
apt:
- libncurses5-dev
- libncursesw5-dev
brew:
- pkg-config
- ncurses
dnf:
- ncurses-devel

# roles/cmatrix/tasks/main.yml
---
- name: cmatrix and dependencies are present
# A var from above, in the playbook
become: "{{ should_be_root }}"

block:
- name: ncurses is present
package:
name: '{{ item }}'
state: latest
# Get an array of the correct package names to install from the map in our
# default variables file
loop: "{{ ncurses[ansible_pkg_mgr] | default(ncurses['default']) }}"

# Install as usual if this is not a brew system
- name: cmatrix is present
when: ansible_pkg_mgr != 'brew'
package:
name: cmatrix
state: present
# If it is a brew system, use this instead
- name: cmatrix is present (brew)
when: ansible_pkg_mgr == 'brew'
homebrew:
name: cmatrix
state: head
install_options: with-some-option
# roles/youtube-dl/tasks/main.yml
---
- name: youtube-dl and dependencies are present
become: "{{ should_be_root }}"

block:
- name: ffmpeg is present
when: ansible_pkg_mgr != 'brew'
package:
name: ffmpeg
state: latest
- name: ffmpeg is present (brew)
when: ansible_pkg_mgr == 'brew'
homebrew:
name: ffmpeg
state: head
install_options: with-chromaprint, with-fdk-aac, with-etc-etc

- name: atomicparsley is present
package:
name: atomicparsley
state: latest

- name: youtube-dl is present
package:
name: youtube-dl
state: latest

The result for Ubuntu:

$ ansible-playbook demo.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [Demo] ********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [cmatrix : ncurses is present] ********************************************
ok: [localhost] => (item=libncurses5-dev)
ok: [localhost] => (item=libncursesw5-dev)

TASK [cmatrix : cmatrix is present] ********************************************
ok: [localhost]

TASK [cmatrix : cmatrix is present (brew)] *************************************
skipping: [localhost]

TASK [youtube-dl : ffmpeg is present] ******************************************
ok: [localhost]

TASK [youtube-dl : ffmpeg is present (brew)] ***********************************
skipping: [localhost]

TASK [youtube-dl : atomicparsley is present] ***********************************
ok: [localhost]

TASK [youtube-dl : youtube-dl is present] **************************************
ok: [localhost]

PLAY RECAP *********************************************************************
localhost : ok=6 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0


Related Topics



Leave a reply



Submit