Cannot Make Bash Script Work from Cloud-Init

Cannot make bash script work from cloud-init

You have specified sh shell under runcmd, but have she-bang set to bash. The latter does not matter because if you run as sh /opt/cloud-init-scripts/whatever.sh it will be run with sh shell. I guess you are probably using a non POSIX shell feature which is incompatible with the sh shell.

Or alternatively if your intention is to run the script in bash shell, change the runCmd in cloud-init script to

runcmd:
 - [ bash, /opt/cloud-init-scripts/whatever.sh ]

cloud-init runcmd (using MAAS)

Anything you run using runcmd must already exist on the filesystem. There is no provision for automatically fetching something from a remote host.

You have several options for getting files there. Two that come to mind immediately are:

• You could embed the script in your cloud-init configuration using the write-files directive:

  write_files:
    - path: /usr/local/bin/foo.sh
      permissions: '0755'
      content: |
        #!/bin/bash
        echo "=========hello world=========" >>foosh.bar
  
  runcmd:
    - [bash, /usr/local/bin/foo.sh]
  

• You could fetch the script from a remote location using curl (or similar tool):

  runcmd:
    - [curl, -o, /usr/local/bin/foo.sh, http://somewhere.example.com/foo.sh]
    - [bash, /usr/local/bin/foo.sh]
  

user-data (cloud-init) script not executing on EC2

Cloud-init does not accept plain bash scripts, just like that. It's a beast that eats YAML file that defines your instance (packages, ssh keys and other stuff).

Using MIME you can also send arbitrary shell scripts, but you have to MIME-encode them.

$ cat my-boothook.txt
#!/bin/sh
echo "Hello World!"
echo "This will run as soon as possible in the boot sequence"

$ cat my-user-script.txt
#!/usr/bin/perl
print "This is a user script (rc.local)\n"

$ cat my-include.txt
# these urls will be read pulled in if they were part of user-data
# comments are allowed.  The format is one url per line
http://www.ubuntu.com/robots.txt
http://www.w3schools.com/html/lastpage.htm

$ cat my-upstart-job.txt
description "a test upstart job"
start on stopped rc RUNLEVEL=[2345]
console output
task
script
echo "====BEGIN======="
echo "HELLO From an Upstart Job"
echo "=====END========"
end script

$ cat my-cloudconfig.txt
#cloud-config
ssh_import_id: [smoser]
apt_sources:
 - source: "ppa:smoser/ppa"

$ ls
my-boothook.txt     my-include.txt      my-user-script.txt
my-cloudconfig.txt  my-upstart-job.txt

$ write-mime-multipart --output=combined-userdata.txt \
   my-boothook.txt:text/cloud-boothook \
   my-include.txt:text/x-include-url \
   my-upstart-job.txt:text/upstart-job \
   my-user-script.txt:text/x-shellscript \
   my-cloudconfig.txt

$ ls -l combined-userdata.txt 
-rw-r--r-- 1 smoser smoser 1782 2010-07-01 16:08 combined-userdata.txt

The combined-userdata.txt is the file you want to paste there.

More info here:

https://help.ubuntu.com/community/CloudInit

Also note, this highly depends on the image you are using. But you say it is really cloud-init based image, so this applies. There are other cloud initiators which are not named cloud-init - then it could be different.

Issues running commands via cloud-init

There are a couple of problems going on here.

First, your cloud config user data must begin with the line:

#cloud-config

Without that line, cloud-init doesn't know what to do with it. If you were to submit a user-data configuration like this:

#cloud-config
runcmd:
  - [ cd, ~ ]
  - [ touch test ]
  - [ echo 'test' > test ]

You would find the following errors in /var/log/cloud-init-output.log:

runcmd.0: ['cd', None] is not valid under any of the given schemas
/var/lib/cloud/instance/scripts/runcmd: 2: cd: can't cd to None
/var/lib/cloud/instance/scripts/runcmd: 3: touch test: not found
/var/lib/cloud/instance/scripts/runcmd: 4: echo 'test' > test: not found

You'll find the solution to these problems in the documentation, which includes this note about runcmd:

# run commands
# default: none
# runcmd contains a list of either lists or a string
# each item will be executed in order at rc.local like level with
# output to the console
# - runcmd only runs during the first boot
# - if the item is a list, the items will be properly executed as if
#   passed to execve(3) (with the first arg as the command).
# - if the item is a string, it will be simply written to the file and
#   will be interpreted by 'sh'

You passed a list of lists, so the behavior is governed by "*if the item is a list, the items will be properly executed as if passed to execve(3) (with the first arg as the command)". In this case, the ~ in [cd, ~] doesn't make any sense -- the command isn't being executed by the shell, so there's nothing to expand ~.

The second two commands include on a single list item, and there is no command on your system named either touch test or echo 'test' > test.

The simplest solution here is to simply pass in a list of strings intead:

#cloud-config
runcmd:
  - cd /root
  - touch test
  - echo 'test' > test

I've replaced cd ~ here with cd /root, because it seems better to be explicit (and you know these commands are running as root anyway).

Using valid YAML in cloud-init to run commands that add text to file

You're insufficiently quoting the values in your list, so the : in your echo statements is getting interpreted as the key: value separator. You want to quote the entire contents of each line, and the best way of doing this is probably using one of the YAML quote operators (>, the folding quote operator, or | the literal quote operator). You'll find some documentation on this topic here.

Like this:

#cloud-config

runcmd:
  - |
    aws s3 cp s3://my-bucket/elasticsearch/elasticsearch.tar.gz /opt/elastic/elasticsearch.tar.gz
  - |
    tar -xf /opt/elastic/elasticsearch.tar.gz
  - |
    ln -sv /opt/elastic/elasticsearch-7.7.6 /opt/elastic/elasticsearch
  - |
    cp /opt/elastic/elasticsearch/config/elasticsearch.yml /opt/elastic/elasticsearch/config/elasticsearch.yml.bkp
  - |
    echo 'cluster.name: DEMO' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'node.name: node1' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'path.data: /opt/elastic/data' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'path.logs: /opt/elastic/logs' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'network.host: host1.domain.internal' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'http.port: 9200' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'discovery.seed_hosts: ["host1.domain.internal", "host2.domain.internal", "host3.domain.internal"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'cluster.initial_master_nodes: ["node1", "node2", "node3"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml

You can verify this parses correctly by feeding it through a
YAML-to-JSON converter, which will show you:

[
  "aws s3 cp s3://my-bucket/elasticsearch/elasticsearch.tar.gz /opt/elastic/elasticsearch.tar.gz\n",
  "tar -xf /opt/elastic/elasticsearch.tar.gz\n",
  "ln -sv /opt/elastic/elasticsearch-7.7.6 /opt/elastic/elasticsearch\n",
  "cp /opt/elastic/elasticsearch/config/elasticsearch.yml /opt/elastic/elasticsearch/config/elasticsearch.yml.bkp\n",
  "echo 'cluster.name: DEMO' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'node.name: node1' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'path.data: /opt/elastic/data' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'path.logs: /opt/elastic/logs' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'network.host: host1.domain.internal' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'http.port: 9200' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'discovery.seed_hosts: [\"host1.domain.internal\", \"host2.domain.internal\", \"host3.domain.internal\"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'cluster.initial_master_nodes: [\"node1\", \"node2\", \"node3\"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n"
]

You should be able to combine those lines into a single multi-line shell script, like this (I've taken the liberty of replacing the multiple echo statements with a single "here"-document):

#cloud-config

runcmd:
  - |
    aws s3 cp s3://my-bucket/elasticsearch/elasticsearch.tar.gz /opt/elastic/elasticsearch.tar.gz
    tar -xf /opt/elastic/elasticsearch.tar.gz
    ln -sv /opt/elastic/elasticsearch-7.7.6 /opt/elastic/elasticsearch
    cp /opt/elastic/elasticsearch/config/elasticsearch.yml /opt/elastic/elasticsearch/config/elasticsearch.yml.bkp

    cat >> /opt/elastic/elasticsearch/config/elasticsearch.yml <<EOF
    cluster.name: DEMO' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
    node.name: node1' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
    path.data: /opt/elastic/data' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
    path.logs: /opt/elastic/logs' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
    network.host: host1.domain.internal' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
    http.port: 9200' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
    discovery.seed_hosts: ["host1.domain.internal", "host2.domain.internal", "host3.domain.internal"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
    cluster.initial_master_nodes: ["node1", "node2", "node3"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
    EOF

That's probably both easier to read and easier to maintain.

Terraform copies but (cloud-init) doesn't run user_data script on Bitnami EC2 instance

I tried to replicate your issue, but your code works perfectly:

      ___ _ _                   _
      | _ |_) |_ _ _  __ _ _ __ (_)
      | _ \ |  _| ' \/ _` | '  \| |
      |___/_|\__|_|_|\__,_|_|_|_|_|
  
  *** Welcome to the Node.js packaged by Bitnami 17.8.0-2                 ***
  *** Documentation:  https://docs.bitnami.com/aws/infrastructure/nodejs/ ***
  ***                 https://docs.bitnami.com/aws/                       ***
  *** Bitnami Forums: https://community.bitnami.com/                      ***
bitnami@ip-172-31-37-75:~$ sudo su -
root@ip-172-31-37-75:~# ls
output.txt
root@ip-172-31-37-75:~# cat output.txt 
Hello World.  The time is now Sat, 02 Apr 2022 22:53:55 +0000!
root@ip-172-31-37-75:~# cd /opt/bitnami/
apache/       bncert/       common/       gonit/        node/         python/       scripts/      var/          
apache2/      bndiagnostic/ git/          nami/         projects/     redis/        stats/        
root@ip-172-31-37-75:~# cd /opt/bitnami/p
projects/ python/   
root@ip-172-31-37-75:~# cd /opt/bitnami/p
projects/ python/   
root@ip-172-31-37-75:~# cd /opt/bitnami/projects/

Thus its possible that your real code that you are using is different then in the question.

---

Related Topics