After restarting nova-api the metadata service came back and the instance is now able to successfully get its user-data but then the process ends when it looks for userData.yml in /tmp/cm/ and it's not there.
[INFO] cm_autorun:78 2015-10-22 13:21:48,638: Getting user data from 'http://169.254.169.254/latest/user-data', attempt 0
[DEBUG] cm_autorun:82 2015-10-22 13:21:48,643: Saving user data in its original format to file '/opt/cloudman/boot/original_userData.yaml'
[DEBUG] cm_autorun:86 2015-10-22 13:21:48,643: Got user data
[INFO] cm_autorun:501 2015-10-22 13:21:48,643: Handling user data in YAML format.
...
[ERROR] cm_autorun:321 2015-10-22 13:23:22,825: Error running boot script '/opt/cloudman/boot/cm_boot.py'. Process returned code '1' and following stderr: 2015-10-22 13:23:22,626 DEBUG cm_boot:21 - Successfully ran 'apt-get update; apt-get install bash'
2015-10-22 13:23:22,627 DEBUG cm_boot:399 - virtual-burrito seems to be installed
2015-10-22 13:23:22,819 DEBUG cm_boot:21 - Successfully ran '/bin/bash -l -c 'VIRTUALENVWRAPPER_LOG_DIR=/tmp/; HOME=/home/ubuntu; . /home/ubuntu/.venvburrito/startup.sh; lsvirtualenv | grep CM''
2015-10-22 13:23:22,819 DEBUG cm_boot:402 - 'CM' virtualenv found
Traceback (most recent call last):
File "/opt/cloudman/boot/cm_boot.py", line 517, in <module>
main()
File "/opt/cloudman/boot/cm_boot.py", line 494, in main
with open(os.path.join(CM_BOOT_PATH, USER_DATA_FILE)) as ud_file:
IOError: [Errno 2] No such file or directory: '/tmp/cm/userData.yaml'
Can anyone shed some light on why this is occurring?
Full cm_autorun.log:
[INFO] cm_autorun:78 2015-10-22 13:21:48,638: Getting user data from 'http://169.254.169.254/latest/user-data', attempt 0
[DEBUG] cm_autorun:82 2015-10-22 13:21:48,643: Saving user data in its original format to file '/opt/cloudman/boot/original_userData.yaml'
[DEBUG] cm_autorun:86 2015-10-22 13:21:48,643: Got user data
[INFO] cm_autorun:501 2015-10-22 13:21:48,643: Handling user data in YAML format.
[INFO] cm_autorun:369 2015-10-22 13:21:48,649: Setting up password-based login for user 'ubuntu'
[INFO] cm_autorun:369 2015-10-22 13:21:48,836: Setting up password-based login for user 'galaxy'
[INFO] cm_autorun:397 2015-10-22 13:21:48,972: freenx-server is not installed; not configuring it
[DEBUG] cm_autorun:575 2015-10-22 13:21:48,972: Composed user data (with the following keys filtered out ['password', 'freenxpass', 'access_key', 'secret_key']): {'storage_type': u'volume', 's3_port': 8080, 'cloud_type': u'openstack', 'initial_cluster_type': u'None', 'cluster_name': u'Biocloud', 'role': 'master', 'bucket_cluster': 'cm-a76621402fef00c9bbf3ee726330c1ce', 'boot_script_path': '/opt/cloudman/boot', 'ec2_conn_path': u'/services/Cloud', 'region_name': u'nova_ec2', 'cidr_range': u'10.117.231.0/25', 'region_endpoint': u'openstack-dev.biodiversity.agr.gc.ca', 'ec2_port': 8773, 'storage_size': u'10', 'cloud_name': u'Openstack', 'boot_script_name': 'cm_boot.py', 'is_secure': False, 'custom_image_id': u'', 'cloudman_home': '/mnt/cm', 'bucket_default': u'cloudman', 'custom_instance_type': u'', 's3_host': u'openstack-dev.biodiversity.agr.gc.ca', 's3_conn_path': u'/'}
[ERROR] cm_autorun:415 2015-10-22 13:21:48,986: Error running 'mountpoint -q /mnt'. Process returned code '1' and following stderr: ''
[WARNING] cm_autorun:439 2015-10-22 13:21:48,986: Mountpoint /mnt not available and no device /dev/xvdb
[DEBUG] cm_autorun:124 2015-10-22 13:21:48,986: Establishing boto S3 connection to a custom Object Store
[DEBUG] cm_autorun:161 2015-10-22 13:22:36,578: Cluster bucket 'cm-a76621402fef00c9bbf3ee726330c1ce' not found.
[DEBUG] cm_autorun:268 2015-10-22 13:22:36,579: Using default bucket 'cloudman'
[DEBUG] cm_autorun:354 2015-10-22 13:22:36,579: Default bucket url: http://s3.amazonaws.com/cloudman
[DEBUG] cm_autorun:294 2015-10-22 13:22:36,579: Could not get boot script 'cm_boot.py' from cluster bucket 'cm-a76621402fef00c9bbf3ee726330c1ce'; retrieving the public one from bucket url 'http://s3.amazonaws.com/cloudman/cm_boot.py'
[INFO] cm_autorun:239 2015-10-22 13:22:36,579: Getting boot script from 'http://s3.amazonaws.com/cloudman/cm_boot.py' and saving it locally to '/opt/cloudman/boot/cm_boot.py'
[DEBUG] cm_autorun:246 2015-10-22 13:22:36,672: Got boot script from 'http://s3.amazonaws.com/cloudman/cm_boot.py'
[DEBUG] cm_autorun:297 2015-10-22 13:22:36,672: Saved boot script to '/opt/cloudman/boot/cm_boot.py'
[DEBUG] cm_autorun:124 2015-10-22 13:22:36,673: Establishing boto S3 connection to a custom Object Store
[DEBUG] cm_autorun:161 2015-10-22 13:23:10,601: Cluster bucket 'cm-a76621402fef00c9bbf3ee726330c1ce' not found.
[INFO] cm_autorun:313 2015-10-22 13:23:10,601: Running boot script '/opt/cloudman/boot/cm_boot.py'
[ERROR] cm_autorun:321 2015-10-22 13:23:22,825: Error running boot script '/opt/cloudman/boot/cm_boot.py'. Process returned code '1' and following stderr: 2015-10-22 13:23:22,626 DEBUG cm_boot:21 - Successfully ran 'apt-get update; apt-get install bash'
2015-10-22 13:23:22,627 DEBUG cm_boot:399 - virtual-burrito seems to be installed
2015-10-22 13:23:22,819 DEBUG cm_boot:21 - Successfully ran '/bin/bash -l -c 'VIRTUALENVWRAPPER_LOG_DIR=/tmp/; HOME=/home/ubuntu; . /home/ubuntu/.venvburrito/startup.sh; lsvirtualenv | grep CM''
2015-10-22 13:23:22,819 DEBUG cm_boot:402 - 'CM' virtualenv found
Traceback (most recent call last):
File "/opt/cloudman/boot/cm_boot.py", line 517, in <module>
main()
File "/opt/cloudman/boot/cm_boot.py", line 494, in main
with open(os.path.join(CM_BOOT_PATH, USER_DATA_FILE)) as ud_file:
IOError: [Errno 2] No such file or directory: '/tmp/cm/userData.yaml'
[INFO] cm_autorun:605 2015-10-22 13:23:22,826: ---> /usr/bin/cm_autorun.py done <---