Blog
November 21, 2017
With so many servers now being virtualized and many of them being pushed to the cloud, automated instance configuration and customization is a must-have. The cloud-init CentOS package is one of my favorite tools for tweaking the settings within an instance — especially if we’re talking about more than a small handful of systems which need to be similarly configured.
In this blog, I'll cover what a cloud-init package is, what it does, how to configure it, and where to get guidance for implementing and maintaing cloud-init on CentOS.
- What Is cloud-init?
- How to Use NoCloud Datasource For cloud-init
- Tips For Using cloud-init CentOS
- CentOS Support For cloud-init Modules
What Is cloud-init?
cloud-init is an Ubuntu package that helps with early initialization of a cloud instance.
What Is cloud-init on AWS?
cloud-init configures specific aspects on AWS.
How to Use NoCloud Datasource For cloud-init
What many folks aren’t familiar with, even those who are familiar with cloud-init is that you can use cloud-init to change the settings on non-cloud-based systems, too! This is known as the NoCloud datasource and uses a vfat or iso9660 filesystem (usually an ISO mounted to a virtual CDROM or a floppy mounted in a virtual disk drive) to deliver the configuration data to cloud-init running on the system. Since the data can be provided by a local filesystem, no network is necessary, making configuration of network-isolated machines consistent and repeatable. The cloud-init documentation for the NoCloud datasource has detailed information and examples.
This is not to say that you can’t store the configuration files on your network if one is available. For instance, you could provide the NoCloud datasource on the kernel command line, and specify that the data is stored locally or is accessible on the network using the HTTP(S) or FTP protocols.
We can now have centralized configuration management, letting us use cloud-init modules to specify the networking, accounts, SSL, disks/partitions, packages and run custom scripts, or utilize more powerful system configuration tools such as Chef or Puppet. Obviously, this NoCloud approach probably won’t scale well to more than a few tens of systems, but if you have a small private or personal cloud, or you want to test your metadata and user data files, NoCloud can be very useful!
Tips For Using cloud-init CentOS
We’ve been using cloud-init internally for over a year and have run into a couple of “gotchas” that you may, or may not, encounter if you use cloud-init on CentOS.
Enterprise Linux Options
CentOS is one of your enterprise Linux options. But you have others. Learn them all in our ultimate decision-makers guide.
1. Addressing Errors
Most recently, version 0.7.9 of cloud-init, which shipped with RHEL/CentOS 7.4, does not play well with NoCloud. If you try to set the system hostname, you will see the following errors in the cloud-init debug output:
Running module set_hostname (<module 'cloudinit.config.cc_set_hostname' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_set_hostname.pyc'>) with frequency once-per-instance
start: init-local/config-set_hostname: running config-set_hostname with frequency once-per-instance
Writing to /var/lib/cloud/instances/iid-ralloway01/sem/config_set_hostname - wb: [420] 18 bytes
Running config-set_hostname using lock ()
Setting the hostname to ralloway.openlogic.local(ralloway.openlogic.local)
Running command ['hostnamectl', 'set-hostname', ralloway.openlogic.local] with allowed return codes [0] (shell=False, capture=True)
Failed to set the hostname to ralloway.openlogic.local (ralloway.openlogic.local)
Failed to set the hostname to ralloway.openlogic.local (ralloway.openlogic.local)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_set_hostname.py", line 47, in handle
cloud.distro.set_hostname(hostname, fqdn)
File "/usr/lib/python2.7/site-packages/cloudinit/distros/__init__.py", line 84, in set_hostname
self._write_hostname(writeable_hostname, self.hostname_conf_fn)
File "/usr/lib/python2.7/site-packages/cloudinit/distros/rhel.py", line 130, in _write_hostname
util.subp(['hostnamectl', 'set-hostname', str(hostname)])
File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1858, in subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['hostnamectl', 'set-hostname', 'ralloway.openlogic.local']
Exit code: 1
Reason: -
Stdout: -
Stderr: Failed to create bus connection: Connection refused
finish: init-local/config-set_hostname: FAIL: running config-set_hostname with frequency once-per-instance
Running module set_hostname (<module 'cloudinit.config.cc_set_hostname' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_set_hostname.pyc'>) failed
Running module set_hostname (<module 'cloudinit.config.cc_set_hostname' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_set_hostname.pyc'>) failed
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 777, in _run_modules
freq=freq)
File "/usr/lib/python2.7/site-packages/cloudinit/cloud.py", line 54, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python2.7/site-packages/cloudinit/helpers.py", line 187, in run
results = functor(*args)
File "/usr/lib/python2.7/site-packages/cloudinit/config/cc_set_hostname.py", line 47, in handle
cloud.distro.set_hostname(hostname, fqdn)
File "/usr/lib/python2.7/site-packages/cloudinit/distros/__init__.py", line 84, in set_hostname
self._write_hostname(writeable_hostname, self.hostname_conf_fn)
File "/usr/lib/python2.7/site-packages/cloudinit/distros/rhel.py", line 130, in _write_hostname
util.subp(['hostnamectl', 'set-hostname', str(hostname)])
File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1858, in subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ['hostnamectl', 'set-hostname', 'ralloway.openlogic.local']
Exit code: 1
Reason: -
Stdout: -
Stderr: Failed to create bus connection: Connection refused
The issue being that cloud-init attempts to talk to dbus before it is started. A workaround, provided in the tickets, may be a solution for some folks, but for me, I didn’t want to modify files provided by the package which would not be replaced when a new version of the package is made available.
Since the version which shipped with CentOS 7.3, cloud-init 0.7.5, was working just fine in my environment, I opted to install that version of the package and lock it with yum’s versionlock plugin when I build the “golden image” that my future instances will be based on:
[root@centos7 ~]# yum -y install http://mirror.centos.org/centos/7.3.1611/extras/x86_64/Packages/cloud-init-0.7.5-10.el7.centos.1.x86_64.rpm
Loaded plugins: fastestmirror
cloud-init-0.7.5-10.el7.centos.1.x86_64.rpm | 418 kB 00:00:00
Examining /var/tmp/yum-root-BFgpYD/cloud-init-0.7.5-10.el7.centos.1.x86_64.rpm: cloud-init-0.7.5-10.el7.centos.1.x86_64
Marking /var/tmp/yum-root-BFgpYD/cloud-init-0.7.5-10.el7.centos.1.x86_64.rpm to be installed
====================================================================================================
Package Arch Version Repository Size
====================================================================================================
Installing:
cloud-init x86_64 0.7.5-10.el7.centos.1 /cloud-init-0.7.5-10.el7.centos.1.x86_64
1.4 M
Installed:
cloud-init.x86_64 0:0.7.5-10.el7.centos.1
Complete!
[root@centos7 ~]#
[root@centos7 ~]# yum -y install yum-versionlock
Loaded plugins: fastestmirror
====================================================================================================
Package Arch Version Repository Size
====================================================================================================
Installing:
yum-plugin-versionlock noarch 1.1.31-42.el7 base 32 k
Installed:
yum-plugin-versionlock.noarch 0:1.1.31-42.el7
Complete!
[root@centos7 ~]#
[root@centos7 ~]# yum versionlock add cloud-init
Loaded plugins: fastestmirror, versionlock
Adding versionlock on: 0:cloud-init-0.7.5-10.el7.centos.1
versionlock added: 1
[root@centos7 ~]#
2. Permission Changes on /etc/ssh/sshd_config
Another issue that I’ve encountered lately is that cloud-init changes the permissions on /etc/ssh/sshd_config to 0644, instead of honoring the existing permissions. This is a known issue with the code in cc_set_passwords.py, or more-so, the write_file() function in util.py, and has been addressed in cloud-init 17.1, but officially, cloud-init 0.7.9 is the newest release available for RHEL/CentOS 7.4. By the way, “cloud-init 17.1” is not a typo. cloud-init was version bumped after 0.7.9 to 17.1.
I encountered this issue when performing PCI-DSS hardening for images that I was creating for a customer. One PCI-DSS test, “Verify and Correct File Permissions” (xccdf_org.ssgproject.content_rule_rpm_verify_permissions), requires that the permissions on all files match the permissions that they were installed with from the RPM. Since the initial file permissions were 0600, cloud-init’s alteration to 0644 triggers a failure for that particular PCI-DSS test.
We can verify the original permissions by querying the RPM database:
[root@centos7 ~]# rpm -qlv openssh-server | grep /etc/ssh/sshd_config
-rw------- 1 root root 3907 Oct 19 17:52 /etc/ssh/sshd_config
[root@centos7 ~]#
We’re now faced with selecting a solution that our mandate and/or business case must support, or at least accept:
- Update cloud-init to 17.1 or newer, which are currently versions newer than that published by the distribution and accept any changes (positive or negative) that come with that version.
- Modify the existing cloud-init 0.7.x package files to maintain or force the 0600 permissions, knowing that these changes may impact future cloud-init package updates.
- Backport the changes made to cloud-init 17.1 which provide the copy_mode option to write_file(), still knowing that these changes may impact future cloud-init package updates.
- Note an exception to the PCI-DSS hardening.
As with most things, the solution that you decide on will depend on your project requirements, environment, and management.
CentOS Support For cloud-init Modules
CentOS may not support all of the modules that cloud-init’s upstream maintainers provide.
When crafting your metadata and user data, it may be beneficial to verify that the modules that you intend to use are supported and, when not supported, determine if you’re willing to use the bootcmd, runcmd, or scripts-* (vendor/per-once/per-boot/per-instance/user) modules to implement the missing functionality.
Comparing the SOURCES/cloud-init-centos.cfg file from the cloud-init 0.7.5 source RPM to the config/cloud.cfg file within the cloud-init-0.7.5.tar.gz archive (included with the source RPM), we can see the following modules are not called:
- seed_random
- ca-certs
- emit_upstart
- disk_setup
- ssh-import-id
- grub-dpkg
- apt-pipelining
- apt-configure
- landscape
- byobu
- scripts-vendor
- power-state-change
- yum-add-repo
Some of the disabled modules make sense, such as emit_upstart, grub-dpkg, apt-*, and landscape, which are tailored towards other Linux distributions (such as FreeBSD, SUSE, Ubuntu, Debian). Others, like ssh-import-id, are disabled when you’d otherwise think that they’d be active.
You will be forgiven for being further confused by the online documentation. If you pull up the cloud-init online documentation, you will be redirected to the latest version (17.1, currently). There are modules supported in 17.1 which simply don’t exist in the CentOS published versions. Unless you change the version using the green “v: ” dropdown at the bottom of the left frame, you will likely be reading docs that don’t apply to you.
I stumbled upon the disabled modules and documentation dissonance when trying to get cloud-init 0.7.4 (or perhaps 0.7.5) to register an instance to our Spacewalk server. Everything looked fine with my metadata and there were no errors in the cloud-init output, but the system refused to register. It turned out that the Spacewalk module was not added until cloud-init 0.7.8, so I used one of the scripting modules to add the Spacewalk registration functionality into the earlier cloud-init version that I had to use. Lesson learned!
Get Support From OpenLogic
OpenLogic provides proven open source solutions for the cloud and beyond.
Our CentOS experts can help you get set up with cloud-init CentOS, migrate from RHEL to CentOS, and so much more.
In fact, when you choose CentOS supported by OpenLogic, you'll get:
- 50% cost-savings.
- Long-term support.
- Backporting.
- Guaranteed SLAs.
- Architectural minimization.
- Multi-platform support.
- CentOS distributions.
- Expert guidance.
- Supported CentOS options in Microsoft Azure and Amazon Web Services (AWS).
Learn more about how OpenLogic experts can help you with cloud-init and CentOS.