Intelligent Password Changes with Puppet

I need to change the root password on all my hosts but I have a small problem: some hosts have older md5 hashed passwords and the newer ones use the more secure SHA-512 hash. If I did not care about the different hashes and wanted to have SHA-512 across the board I would do a very simple manifest entry to make this happen: Problem is I want to replace the old md5 hashes with new md5 hashes and the old SHA-512 with new SHA-512; not something that Puppet supports very easily. To do this we are going to build a new module with a Custom Fact written in Ruby. First off I need to explain some things if you are new to Puppet.

• A module is stored under /etc/puppet/modules and is called via an include declaration in the site.pp Master Manifest
• A module has it's own Manifest called init.pp under /etc/puppet/modules/<module_name>/manifests
• Inside the init.pp is a class that MUST be named the same as your folder structure. Example: if the folder at /etc/puppet/modules/<module_name> is named "rootpass" then your class declaration must be "class rootpass {...."
• A module is very powerful and the functionality is written in Ruby

Now that we have those out of the way lets start. If you are doing this your self here is the folder structure to make life easier:

Let's visit each component a piece at a time

1. Custom Fact - sha512rootpass.rb. Puppet if Statements can be a bit tricky (see http://docs.puppetlabs.com/learning/variables.html) so in this case I needed a Custom Fact that checked to see if the root password was a SHA-512 hash which is indicated by it starting with "$6$" (md5 is $1$). If the root password is indeed a SHA-512 hash then the variable sha512rootpass will return with a "true" value. This functionality is delivered by Ruby Facter. For more information take a look at http://docs.puppetlabs.com/guides/custom_facts.html. My custom fact is silly simple, it just greps the shadow file for "root:$6$*" and if it's there then returns "true" which means that root has a SHA-512 hashed password.

2. New Class - init.pp. The logic for the operation that we actually want to run is located in the init.pp file. Here is where we define our class (reminder, it needs to be the same as you top level folder name). This one basically says "If root is using a SHA-512 password hash (defined by $sha512rootpass = true) replace it with this new one. If not then assume it's md5 and replace it with this new md5 hash.

Now we need to tell Puppet what servers to apply this to and this is done by modifying the site.pp Manifest on the Puppet Master. For now I'm going to apply it to all my nodes and so I just add it to my default. If you wanted you can add a new section that says "node <hostname> {include rootpass}" and it would be applied just to that host.

Now lets test it on an agent box that has an md5 hashed password and a box that has a SHA-512 password. Our older box with md5 is the first up to bat.....

As you can see the password was a md5 ($1$) and was changed appropriately. Next let's look at a box with SHA-512.

As you can see the old password was a SHA-512 hash and has been replaced with the new SHA-512 hash. Success!

Getting Started with Puppet Open Source

I'm starting to work with Puppet and noticed that when I am using the open source version there is not really a good "Getting Started" guide and documentation is rather lacking. Not wanting anyone else to suffer through that here is my attempt at it. Hope it helps others.

Building the Puppet Master

First we are going to check what OS we are running, in my case it's CentOS 6.4 x64 so we're going to grab the repo from yum.puppetlabs.com. After that I look to see what's available and then finally install with a yum install puppet-server.noarch.

 

Once Puppet is installed we need to do some things. First we take a look at the stock puppet.conf file. Now, let's make it useful by adding the server name (remember: this is the Puppet Master so it's $HOSTNAME) and enabling pluginsync.

As you can see the file structure of Puppet is pretty empty with the open source version... Let's add site.pp which is the master Manifest for all your Puppet tasks.

I'm going to add a very simple puppet command that applies to all hosts (nodes) and creates a new user with a SHA-512 password hash. This Manifest file is the source of truth for all your Puppet tasks, more about that later for now remember site.pp is critical.

Lastly before starting the services we need to open 2 ports on the firewall.

Now that the Manifest is complete and we have open ports let's start the Puppet and Puppet Master services.

We did all that work, let's see if Puppet works. (Note, if you get an error here there is a good chance that iptables is blocking your Puppet traffic). To do that we are going to call the Puppet agent and tell it to run, but not apply any changes (-noop). As we can see it detected that our new user account is missing but did not change anything.

That's all great, now let's apply it. There are a couple ways to do this:

1. Wait 30 minutes, the agent will automatically run and apply the change.

2. Run puppet agent --test

As we can see, the change was successfully made on the Puppet Master, now lets go start installing an agent on another host.

Installing Puppet Agents

As you can see we are doing basically the same thing as on the Puppet Master but only installing the Puppet Agent.

After the install completes we need to configure the agent to talk to the Puppet Master. This configuration is done in the same /etc/puppet/puppet.conf as the Puppet Master but we change what we add...

Next you need to start the Puppet Agent Service: puppet resource service puppet ensure=running enable=true

At this point assuming no firewall issues your agent is now talking to the Puppet Master (test using the "puppet agent --test --noop" command we used earlier); however there is still one thing that needs to be done. We need to approve the agent's certificate on the Puppet Master; once that is complete then the agent will start applying changes that are specified in the Puppet Master's site.pp. You do that from the Puppet Master using the puppet cert commands:

Congratulations! You just setup a Puppet Open Source instance and are now well on your way to using Puppet to help you manage your infrastructure.

IPv6 Regex

I needed to do a massive rip and replace on some IPv6 IP’s and so a regex seemed the best way to go.

What I was using: Link-local

fe80::([0-9a-f])*:([0-9a-f])*:([0-9a-f])*:([0-9a-f])*

All IPv6 IP’s.

([0-9a-f])*::([0-9a-f])*:([0-9a-f])*:([0-9a-f])*:([0-9a-f])*

Clustering vCenter Orchestrator 5.5 using PostgreSQL.

It’s funny, I’ve edited this post 4 times because I ran into little catch-22 situations as I continued to work using my test vCO instance. Hopefully this post will save somebody else some time when configuring vCO 5.5 in a cluster. Let’s get started!

Deploy a new VM that will host the PostgreSQL Database. I’m using CentOS just in case you are curious.

You can find the latest version of PostgreSQL with:

yum list postgres*

Now install PostgreSQL-server:

yum install postgresql-server

Once it is done installing then we need to configure Postgres:

chkconfig --level 2345 postgresql on

service postgresql initdb

vim /var/lib/pgsql/data/postgresql.conf

Un-comment and modify the listen_addresses and port:

Next modify the what servers are allowed to talk to the Postgres database and how. The database and user have not been created yet but we are going to call them vco and vcouser. The method is an md5 hash of the password for authentication.

vim /var/lib/pgsql/data/pg_hba.conf

Now start Postgres and create the vCO user and database:

service postgresql start

su postgres

psql

CREATE USER vcouser with PASSWORD '$uperG00dP@ss!';

CREATE DATABASE vco;

GRANT ALL PRIVILEGES on DATABASE vco to vcouser;

\q

Now I’m going to deploy 3 vCO appliances from OVA. Near the end of the process to deploy the OVA you will be prompted to create a password. The second password's username is vmware and it is for the web interface that you will use in a minute to configure vCO.

Go to to http://<primary vCO node IP> and click on Orchestrator Configuration. From here login using vmware for the username and the password that you specified when you deployed the OVA.

Next go to the Database section and fill in the information for the PostgreSQL database server we just built. It should fail with an error that the database needs tables created. Click to create.

Once you click to create the tables then it's time to generate self-signed certificates. Navigate to the Server Certificate section and chose the self-signed option. Give it the FQDN of your VIP. (Example: I have cos-test-vco1, cos-test-vco2 and cos-test-vco3 but the VIP is cos-test-vco)

Now we need to grab our vCenter Server's SSL Certificate. Click on Licenses and SSL Certificates

Give it https://<IP of vC Server> and verify the import.

Add any plugins that you want (they can be found at https://solutionexchange.vmware.com).

Next navigate to the Licenses section and give it the IP of your vCenter. Once this succeeds you should have all green statuses.

It's now time to configure vCO to work in a cluster. Go to Server Availability and change it to have 2 active nodes:

Next navigate to vCenter Server and add a new vC. BEWARE: the default setting is "Session per user" and this will appear to succeed on this screen but will be broken later on down the road if you don't change it. The only reason this should be left at the default is if you are using SSO and the same user has rights on both vCO and vCenter Server.

Now because of the change going from the internal to the external Postgres database we need to reinstall all the plugins even though they show up as green on the configuration screen. Don't believe me? If you continue as-is you will run into the below screenshot where your workflow elements are gone.

To do that click "Reset Current Version" and reboot the VM.

Lastly we need to modify the network binding to the correct IP address:

Once you have reached this point it's time to export the vCO Configuration. This is critical because all of your vCO servers in the cluster must be identical with the exception of the network binding. Copy this file via SCP to the other Orchestrator VMs.

Next repeat the below steps for each additional vCO server.

Import the vCO Master Node's configuration file making sure to UNCHECK the override box.

Configure networking to the correct IP address for each node.

At this point you should get an error that says that vCenter is not configured correctly. Follow the prompts and re-enter your credentials to connect to the vC Server.

Once this is completed start your Primary Node and wait for it to start, then start all the other nodes.

Repeat for all nodes until they show as online.

Congratulations, you have now configured a vCO Cluster with a standby node.

VMworld 2013 Hands On Labs!

Here’s a little preview of one of our two OneCloud datacenters that is running VMworld 2013 Hands On Labs. We’re using Hyperic, vC Ops and VMware Log Insight to make sure that you guys have the best labs possible! Enjoy!

Check out these awesome custom vC Ops Dashboards!

This one displays vCD information such as the current VM Consoles that each cell server is serving to clients. Most of these metrics are gathered via a custom Hyperic vCD Plugin that we built (to be posted later).

Simple but effective Lab and VM deployment stats and trending.

Shout out to my co-worker Jacob Ross who was my teammate in designing and building the monitoring for VMworld 2013 HOL. Hope you all enjoy the show!

Monitoring vCD 5.1 vAPP deployment times with SQL

Whipped up a quick set of SQL scripts that will allow me to monitor vAPP deployment times in vCloud Director 5.1. Maybe somebody out there will also find them useful.

--Finds the deployment and vAPP creation times over a set period of time which is currently 6 hours.
select distinct
     Jobs.operation
    ,Jobs.object
    ,Jobs.Task_Length_Minutes
    ,COUNT(Jobs.job_id) as VM_Count
    ,Jobs.Minutes_Since_Task
    ,Jobs.OrgVDC
from (
      select top 2500
         jobs.job_id
        ,jobs.starttime
        ,jobs.stoptime
        ,jobs.object
        ,jobs.operation
        ,DATEDIFF(MINUTE,(jobs.starttime),(jobs.stoptime)) AS Task_Length_Minutes
        ,DATEDIFF(MINUTE,(jobs.starttime),getdate()) as Minutes_Since_Task
        ,org_prov_vdc.name AS OrgVDC
     from jobs
         Join vapp_vm on vapp_vm.vapp_id = jobs.object_id
            JOIN vm_container on vm_container.sg_id = jobs.object_id
            JOIN org_prov_vdc on org_prov_vdc.id = vm_container.org_vdc_id
        WHERE jobs.operation IN (
                  'VAPP_DEPLOY'
                  ,'VDC_INSTANTIATE_VAPP')
            AND DATEPART(YEAR, stoptime) <> 9999
            AND DATEDIFF(MINUTE,(jobs.starttime),getdate()) <=360
               Group by jobs.object, jobs.starttime, jobs.stoptime, jobs.operation, vapp_vm.name, jobs.job_id, org_prov_vdc.name
      ) Jobs
Group BY  Jobs.operation
    ,Jobs.object
    ,Jobs.Task_Length_Minutes
    ,Jobs.Minutes_Since_Task
    ,Jobs.OrgVDC
    Order by Minutes_Since_Task      


--Finds the Average Deployment Time by vAPP Name over a set period of time which is currently 6 hours.
SELECT Jobs.object
        ,AVG(Jobs.TaskLengthSeconds) as AverageDeployTime
        ,Jobs.ElapsedTimeMinutes
FROM (SELECT top 500 jobs.object
,jobs.starttime
        ,jobs.stoptime
        ,DATEDIFF(MINUTE,(jobs.starttime),(jobs.stoptime)) AS TaskLengthSeconds
        ,DATEDIFF(MINUTE,(jobs.starttime),getdate()) as ElapsedTimeMinutes
     FROM jobs
        WHERE (jobs.operation = 'VAPP_DEPLOY' or jobs.operation = 'VDC_INSTANTIATE_VAPP') and DATEPART(YEAR, stoptime) = 2013
            Group by jobs.object, jobs.starttime, jobs.stoptime
                Order by jobs.starttime DESC) Jobs
WHERE CAST(ElapsedTimeMinutes AS int) <=360
    Group by Jobs.object, Jobs.ElapsedTimeMinutes
        Order by Jobs.ElapsedTimeMinutes

       
--Finds the Longest Deployment Time by vAPP Name over a set period of time which is currently 6 hours.
SELECT Jobs.object
        ,MAX(Jobs.TaskLengthMinutes) as MaxDeployTime
FROM (select top 2500 jobs.object
        ,jobs.starttime
        ,jobs.stoptime
        ,DATEDIFF(MINUTE,(jobs.starttime),(jobs.stoptime)) AS TaskLengthMinutes
        ,DATEDIFF(MINUTE,(jobs.starttime),getdate()) as ElapsedTimeMinutes
     FROM jobs
        WHERE jobs.operation = 'VAPP_DEPLOY' or jobs.operation = 'VDC_INSTANTIATE_VAPP' and stoptime not like '%9999%'
            Group by jobs.object, jobs.starttime, jobs.stoptime
                Order by jobs.starttime DESC) Jobs
WHERE CAST(ElapsedTimeMinutes AS int) <=360
    Group by Jobs.object           
        Order by MaxDeployTime

Updating, Replacing or Downgrading Hyperic System Plugins

Sometimes you have an updated Hyperic plugin that you need to replace with a different version on an agent and it will not push for various reasons. To get around this just copy the new plugin file directly to /opt/hyperic/hyperic-hqee-agent/bundles/agent-5.7.0/pdk/plugins on the agent machine and restart the agent service (service hyperic-hqee-agent restart). At this point you are good to go.