Installing Openshift Origin (OKD) on AWS – part 3: scale up your brewery!


In the previous two sessions of this blog series we’ve created an AWS instance and used Ansible to install a single Openshift OKD node. As scalability is one of the key features of cloud, we’ll show you how to add another AWS instance to your cluster. To achieve this, we’ll use Ansible to expand the Kubernetes cluster, without any downtime for your pods.

In this blog post Jan van Zoggel and I focused on installing an all-in-one OKD cluster on a single AWS instance and expanding the cluster with a second AWS instance and manually expanding the cluster with a second AWS instance. This scenario was used for the Terra10 playground environment and this guide should be used as such.

In the previous two blogposts we’ve looked at installing Openshift on a single AWS instance, which is great to get started with Openshift. But what if your developers get excited and start launching lots of Pods? You might hit some performance issues, as this node is also used for the ‘Master’ role which includes hosting the Web console and API’s. Additionally, your master node runs Prometheus and Grafana monitoring out of the box. This is great to get some performance insights but unfortunately those components grab a fair amount of memory.

In this final blog post, we’ll show you how to expand the cluster with a dedicated ‘compute’ node using an additional AWS instance.  Compute nodes are designed to take care of the custom pods your developers create, so you can dedicate your resources on those. By adding multiple compute nodes, you can even ‘evacuate’ Pods from one node to the other in case of maintenance or issues. For now we’ll just focus on one compute node.

To add a second AWS instance, log in to AWS console, hit the ‘launch instance’  button and select Centos 7 from the marketplace. For the instance size the requirements dictate you should have at least 8GB for the compute node, so t3.large would be fine. For the remainder of the wizard you can use the same settings as described in part 1, but be sure to select the same subnet from the dropdown. Also you need to be certain that you’re adding a second disk for Docker. If you’ve added all the ports in the security group as described in part 1, you can select it from the dropdown during the wizard.

Once you have finished the wizard, it shouldn’t take long for the second AWS instance to get a public IP address. Use your AWS key to log in with SSH as we’re now going to preconfigure the host. You could of course re-run all the steps described in part 1 of the blog series, but to make life a little easier, you can also use a script I’ve prepared to run all the preparation steps on Linux:

sudo yum install git
git clone
cd prepareOKDNode

The script installs all packages, configures docker and installs the EPEL release. You’ll find the logfile in /tmp containing all steps. If you hit an error, please type ‘export DEBUG=TRUE’ to print additional logging information.

To grant access from the masternode, please log in to both instances with your AWS key and add the contents of from the masternode to the file ‘authorized_keys’ on the newly created node. This step is essential, as Ansible needs to access all hosts in the cluster without prompting for passwords. Also, note the internal AWS hostname for your newly created instance by typing:


in the SSH session of your new host. You’ll see something like

This is the internal hostname that we can add to our inventory file. Head on over to your master node using SSH and add the following at the end of your inventory file:

[new_nodes] openshift_node_group_name='node-config-compute' '

Once you’ve added the new node to the inventory file, run the following Ansible playbook:

cd ~/openshift-ansible
ansible-playbook ./playbooks/openshift-node/scaleup.yaml

You’ll see that Ansible tries to connect to the second AWS instance and after that several basic pods (for monitoring, the Software Defined Network and some other infra components) are installed onto the new node. When the playbook is done, enter the following on the master node:

oc get nodes

The output should provide you with a list of two nodes, both having the status ‘Ready’. Also, notice that the defined role for the second node is now ‘compute’


As both nodes are now schedulable, your pods will run on either the first or second node. To find out which pods are running on your master node, run

oc describe node <masternode>

Here you’ll see a lot of information, but two things are noticeable here. First, you’ll see that this node is considered ‘schedulable’, which means that new pods are allowed to run on this node. Secondly, you’ll get an overview of all the pods running on your master node. Beside all the default and Openshift pods, you might find some other pods which belong to your developers. If you’ve used part two of this blog series to get here, you’ll find the beerdecision pod on the list.

Of course you want your developers running their pods on the new compute node only, so we’ll have to prevent new pods to hit your master node. Additionally you need to evacuate the pods from the master. Lets start with this first step, by marking our master node as unschedulable:

oc adm manage-node <masternode> --schedulable=false

That’s it! By setting the ‘schedulable=false’, the replication controller knows to avoid this node. You can verify this by hitting the ‘oc get nodes’ command again, which will show you the new status for the mater node.


Now we’re going to the existing pods around to the new compute node. Lets assume you’ve set up the project described in part 2, simply switch to this project by typing:

oc projects t10-demo

now, see which label is attached to our beerdecision pod:

oc get pods --show-labels

Your output will look something like this:

[centos@brewery ~]$ oc get pods --show-labels
NAME                   READY     STATUS      RESTARTS   AGE       LABELS
beerdecision-1-build   0/1       Completed   0          17h
beerdecision-1-wtvr6   1/1       Running     0          17h       deployment=beerdecision-1,deploymentconfig=beerdecision,name=beerdecision

You can see our running beerpod has the label ‘name=beerdecision’ among other labels. Let’s move this pod to the other node!

oc adm manage-node <masternode> --evacuate --pod-selector=name=beerdecision

And there we go! Your pod is created onto another available node (which in our case can only be the compute node) and it’s removed from the master node. Because Kubernetes uses a Software Defined Network, you don’t need to reconfigure anything on the network level. All pod traffic is handled by the Openshift Router pod.

As you might notice, our master node still has the ‘compute’ role when we type ‘oc get nodes’. To remove this compute role, simply remove the label from the node

oc label node <masternode>

The minus at the end tells Kubernetes to remove the entire label. When you now enter ‘oc get nodes’, you no longer see the compute role defined at your masternode.


And that’s it! All your new pods will be scheduled on the new Openshift node we created during this blog. We’ve also shown how to move your existing beerpod to the new node without downtime. Although this is just a small brewery environment with a limited amount of beer(pods), we’ve tried to demonstrate the scalability of Openshift on the cloud. Using these basic concepts, you can serve as much beers as needed!


Installing Openshift Origin (OKD) on AWS – part 2: How to make a beerdecision


In this blog series we’re installing Openshift Community Edition (known as OKD, I don’t get it either) on AWS instances based on Centos. In the previous blog post, we’ve set up the AWS instance and used several steps to preconfigure the host. In this part we’ll run the actual Openshift installation using Ansible. To verify that the domain is running, we’ll deploy and scale a simple web application using the web console.

In this blog post Jan van Zoggel and I focused on installing an all-in-one OKD cluster on a single AWS instance and expanding the cluster with a second AWS instance and manually expanding the cluster with a second AWS instance. This scenario was used for the Terra10 playground environment and this guide should be used as such.

The first step towards installing the Openshift cluster is grabbing all the install scripts. Because OKD is open source, you can simply grab these from the official Github page. We’re going to put them in /home/centos/openshift-ansible:

cd ~
git clone
cd openshift-ansible
git checkout release-3.11

It’s important to checkout the branch of the version you wish to install, otherwise you’re using the scripts that are currently in development and you’re likely to run into some bugs. We’re using OKD 3.11 which is the latest release at this time (based on Kubernetes 1.11) but if you want to install a different version, just checkout that branch.

Ansible uses several playbooks to install your (single node) cluster, but it also requires an inventory file which contains all your desired hosts and settings. Creating the inventory file can be a bit challenging because there are a lot of variables to choose from. Although there are many online examples, you should be cautious because the available variables change with each OKD release. For example, to set a node as an ‘infra’ node you had to set the ‘region’ as ‘infra’ using the ‘openshift_node_labels’ variable. In the latest release this has changed completely, as you now have to use the ‘openshift_node_group_name’ variable and set it to ‘node-config-infra’. You can find the latest list of variables here, and the blogposts still help to determine some example values.

I’ve used the following inventory file to create the single node domain, and we’ll go through each of these values in more detail below. Replace the file in /etc/ansible/hosts with the following:

# Create an OSEv3 group that contains the masters, nodes, and etcd groups

# Set variables common for all OSEv3 hosts

# using htpassword authentication on master
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]

# disable memory checks on the EC2 host

# set the default subdomain

# host group for masters

# host group for etcd

# host group for nodes, includes role info
[nodes] openshift_node_group_name='node-config-all-in-one'

Although you can set many more variables, these covers the basics. The file is divided in several sections, as indicated by the brackets for each section. Let’s start at the OSEv3:vars section.

Ansible_ssh_user is set to either root or a user that belongs to the wheel group, in case of our AWS image we use ‘centos’ as this user is allowed to execute everything out of the box. When you set the ansible_become variable, you tell Ansible that it needs to use ‘sudo’ to run commands. Openshift_deployment_type is set to Origin, as we use the open source version of Openshift.

In our playground environment we’re not going to fuss around with LDAP, so we’d like openshift to authenticate against a local htpasswd user. Simply set the openshift_master_identity_providers variable to the specified value, we’ll add the actual user after the installation. Next, we use openshift_disable_check to prevent Ansible to check the memory and disks requirements. This can be useful if you chose the t3.medium instead of the t3.xlarge for budget reasons. The final variable in this section is openshift_master_default_subdomain, which tells Kubernetes that apps run at [projectname]-[appname] It’s very important that this value matches the wildcard cname you used in the DNS configuration in part 1.

The masters section lists all AWS instances used for the master role in your Openshift cluster. Note that you can set up multiple masters (at least 3 for High Availability), but this involves setting up several other variables and a loadbalancer host. Note that this variable also contains the openshift_public_hostname attribute, which matches the other cname you’ve set up in  the DNS at part 1.

The etcd section in this case points to our master host, as it’s common to co-locate our etcd with our masters. At the ‘node’ section, you specify all your nodes (masters, infra nodes and compute nodes combined) and you use the openshift_node_group_name attribute to specify whether a node has a master, infra, compute or all-in-one role. As we only have one EC2 instance, we’ll pick the latter.

Once you’ve got the inventory file set up, you can start with the prerequisite Ansible playbook. This playbook checks if your config file makes sense and checks the hosts in the files to verify that all required components are set.

cd ~/openshift-ansible
ansible-playbook ./playbooks/prerequisites.yml

Ansible will look at /etc/ansible/hosts by default, use -i to specify another location. Don’t worry if you hit an error while running the playbook, the prerequisite playbook (and the installation playbook to some extend) is rerunnable so you can modify your inventory file and repeat the playbook until you hit the desired result.

The prerequisite playbook should verify that all hostnames and URLs in your inventory are resolvable but be sure that it uses the DNS to achieve this! If you put hostnames in ‘/etc/hosts’ you’ll fool the Ansible scripts, but /etc/resolve.conf is copied inside the containers. As a result, the containers will be unaware of the hostnames and this will cause a fatal error in the installation part. If you’ve got all cnames in the DNS as described in part 1, you should be fine.

If your prerequisite completes you can run the installation playbook:

ansible-playbook ./playbooks/deploy_cluster.yml

This may take a while to complete, as Ansible will now install Kubernetes, create an internal registry, set up multiple internal pods and you’ll even get Prometheus and Grafana monitoring out of the box.

Once you’re done, verify that your ec2 instance is now running as an Openshift node with access to the API by typing the following command:

oc get nodes

This output shows you a couple of things, mainly that:

  1. The oc (or kubectl) command now logs in to your master and can be used to access the cluster.
  2. Your node has status ‘Ready’ and owns the compute, infra and master roles.


Before we can open a browser and access the webconsole, we need to set up the user to login to the cluster. For this playground environment, we’ve specified the htpasswd_auth in the inventory file. Simply set up the admin user like this:

sudo htpasswd -c /etc/origin/master/htpasswd admin

You’ll be prompted to insert a new password. Although this is just a playground, please pick a strong password as this is used to access your public web console. After that, just head over to your web console using the URL you’ve set up in the inventory file! In the example above, this would be:

You might hit a certificate error as you haven’t set up certificates yet for this domain. Just ignore the alert and head over to the service catalog.

In here, we’re going to create our first project. In the upper right corner, press the blue  ‘Create Project’ button and enter your project name. Hit the Create button when you’re done.


Next, click your project from the right pane and you’ll automatically go to the dashboard page of your new project. Hit the blue ‘Browse Catalog’ button to add a template.


Pick ‘Nginx HTTP server and reverse proxy’ from the catalog. In the dialog screen, click Next. As project name, we’ll use ‘beerdecision’. You can set a lot of settings in this page, but for now we only focus on the Git Repository URL and the Context Directory. Point the Git Repository to:

and set your Context Directory to:


That’s all we need, hit next and ignore the binding for now. As you hit ‘Create’, Openshift will connect to Github and the source code for a small React web application is downloaded into a new Nginx container. This might take a few seconds.


It won’t take long before you see your Pod appear. The blue circle indicates that the Nginx container is ready to handle the traffic. You can use the up and down arrows to increase or decrease the amount of containers Openshift uses to handle the traffic.


If you’ve used the examples in this guide, your React app should now be available on the following endpoint:

The app will help you decide what beer to drink! You can add options and the random selection will change as well.


Pretty cool right? But what if the Git code changes? If the developer decides to make a code change, you can simply hit the ‘start build’ button using the hamburger icon on the right and Openshift will download the latest code from Github into a second Nginx image. Next, all traffic will automatically move to the new pod without downtime for the app users!


So that’s it for this section on the installation of Openshift. We’ve set up a single node cluster on our AWS instance and we’ve even deployed our very first code into a project. In the last section we’re going to add a second AWS instance and we’ll set it up as a compute node to handle all application pods. Stay tuned for part 3!

A special thanks to Marcel Maas for providing the React Git repository for this guide.

Rubix is a Red Hat Advanced Business Partner & AWS Standard Consulting Partner

Installing Openshift Origin (OKD) on AWS – part 1


Openshift is gaining in popularity, judged by the increasing number of large companies considering Openshift Enterprise as their new container platform. Even though Red Hat offers Openshift as a cloud service, and you can also install Openshift Enterprise on-premise on RedHat Enterprise Linux 7, for this blog post I decided to get my hands dirty with the open source side of the platform. As this guide will show, you can easily enjoy the power of the container platform without any servers or license subscriptions of your own. In a series of three blog posts, we’ll look at installing the open source edition of Openshift, also known as OKD. As open source Linux we picked Centos 7, which is similar to RHEL7.To sweeten the deal, we’ll use the AWS cloud to scale our platform.

In this blog post Jan van Zoggel and I focused on installing an all-in-one OKD cluster on a single AWS instance and expanding the cluster with a second AWS instance manually. This scenario was used for the Terra10 playground environment and this guide should be used as such.

As most readers already know, containers changed the way we release and run applications. There are several benefits to running your applications in containers, but managing and scaling them can be difficult as the number of containers and the amount of docker images grows. Openshift tries to manage these containers with Kubernetes (which is called K8s by the cool kids): the container management layer based on Google’s design. On top of that, Openshift adds several tools such as a build-in CI/CD mechanism, Prometheus monitoring and a nice webconsole if you’re not a fan of the build-in CLI.

For developers that want the basic Openshift experience without the hassle of installing an entire cluster, you can easily install minishift (or minikube if you want to stick to K8s only). In fact, we’ve got some labs to get you started. However, if you’re interested in the installation on multiple hosts so you can play around with High Availability (and you don’t mind diving into the various infrastructure challenges) you’ll have to do an installation.

In blog part 1, we’ll briefly look at setting up AWS instance with Centos 7. Also we’ll look at installing the appropriate packages, configuring Docker Storage, setting up SSH keypairs and setting up the DNS configuration using AWS Route53.

In part 2, we’ll create an inventory file and run the “prerequisite” and “install” Ansible playbooks to set up an all-in-one singlenode cluster. Next, we create a simple user and test with a basic project.

Part 3 of the blogseries will show you how to (manually) configure a second AWS instance, scale out the cluster using a playbook, and move your pods to this second host without any network or DNS changes. (Using the AWS Scaling Group would allow you to add more hosts to your cluster as your load grows, but this is out of the scope of this blog.)

Let’s get started!

Setting up an AWS instance

Creating an instance in AWS is easy. We could use AWS CloudFormation to set up everything as Infra-as-Code but for now we learn more by doing it manually. So just hit the ‘launch instance’ button in the AWS console (assuming you have everything set up), search for Centos in the AWS Market place and choose CentOS 7 from the list. Note that you can also use a RHEL 7 image for this, but additional charges will apply. The official requirements for masters and nodes are pretty high, but you can override the check on this in the installation as we’ll describe in Part 2. For our playground we used t3.medium despite the 4-core minimum warning for co-located etcd and which might also cause some memory-related performance issues. Use t3.xlarge if you want to meet the official requirements.

In the instance detail screen, select a subnet from the dropdown (as your other future instances will have to run in the same subnet) and make sure you have auto-assign Public IP turned ON. All the other settings are adjustable to your own AWS preference  and spending budget, but keep in mind to set a primary IP in the subnet range at ‘eth0’ below. This is just an internal IP used by AWS instances to talk to any future instances you might want to add, as you’ll see in Part 3.

In the ‘Add Storage’ section you get 8gb by default for the root filesystem, though 40gb is required as /var/lib will grow a lot. Also, you should add a second block device with 50gb diskspace for docker storage. In step 6 you can create a security group, default port 22 is enabled for SSH. For security reasons you can select ‘My IP’ in the source dropdown, this way only ssh connections from your current location are allowed. To enable communication with future AWS instances you can add all documented ports to this security group. Note that it’s best practice to use the internal subnet you defined at eth0 as the ‘source’ for these.

When you hit the ‘launch’ button at the end of the wizard you’ll get a PEM key which you can use to connect to your new created instance. Keep this file in a safe spot, as anyone can use it to connect to your instance! After a while you’ll see the public DNS entry appear at your EC2 instance. You’ll need this for the DNS setup. Please note that this DNS entry will change if you shut down your instance.

Setting up the DNS

In this scenario we have one single node to handle all our traffic so this makes our DNS setup fairly easy. In AWS Console, go to Route 53 and set up a hosted zone (e.g. In this hosted zone, add two cname records:

  1. ‘brewery’ (which points to your public DNS entry in the paragraph above)
  2. ‘*.apps.brewery’ (which in this case points to the same public DNS entry as ‘brewery’)


Using the above example cnames, your webconsole will be available at:
and your pods will run at

Setting up the packages on the host

Next, log on to the host using your new SSH key. On Linux and MacOS you can simply run ssh -i [your-key-file] Note that the centos user comes out of the box when picking the centos image from the AWS marketplace.

Once you’re logged in, run the following commands:

sudo yum -y install centos-release-openshift-origin etcd wget git net-tools bind-utils iptables-services bridge-utils bash-completion origin-clients yum-utils kexec-tools sos psacct lvm2 NetworkManager docker-1.13.1

These will install all required packages on Centos which are mentioned on the prerequisites page of OKD, as well as several missing packages on the Centos 7 image of AWS. NetworkManager should be enabled to be available after reboot:

sudo systemctl enable NetworkManager

Setting up Docker

We installed Docker, but to set up the second disk as docker storage device we need to first find the block device name using lsblk:

​[centos@brewery ~]$ lsblk
nvme0n1 259:0 0 40G 0 disk 
└─nvme0n1p1 259:2 0 40G 0 part /
nvme1n1 259:1 0 50G 0 disk

As the example shows, the empty block device is called ‘nvme1n1’, so we can add easily add it to the Docker storage configuration file:

sudo cat < /etc/sysconfig/docker-storage-setup

Next, run the Docker storage setup. This will use the disk you’ve added in the ‘DEVS’ section to create a volume group called ‘docker-vg’:

sudo docker-storage-setup

You can verify that the volume group was created by running ‘sudo vgdisplay’.

Docker is almost ready, we just need to add the following part to the ‘OPTIONS’ line in the /etc/sysconfig/docker file:


For your final step, make sure Docker starts at boot time:

sudo systemctl enable docker

Setup and prepare Ansible

Ansible is not available in the default repository of Centos, so we need to add the EPEL repository. EPEL is short for ‘Extra Packages for Enterprise Linux’ and we can easily add this to Centos with a single command:

sudo yum -y install

However, we don’t want to use EPEL for all packages, so we’re going to disable the repo file. For the installation of Ansible, we explicitly state that we need EPEL for this package:

sudo sed -i -e "s/^enabled=1/enabled=0/" /etc/yum.repos.d/epel.repo
sudo yum -y --enablerepo=epel install ansible pyOpenSSL

Generating a SSH-keypair

Ansible uses SSH to connect to all hosts in your inventory file, even if you’re only using a single host. To achieve this, you’ll have to set up a SSH-keypair for the centos user so login works without passwords. You already have a SSH-keypair from AWS, but it’s not recommended that you use this! Placing your private (generic) AWS keys on multiple locations is a security hazard. And besides, setting up a separate SSH-keypair is easy:

ssh-keygen -b 4096

Accept all defaults and you’ll find the id_rsa and files in the /home/centos/.ssh folder. Next, add the to authorized_keys:

cat ~/.ssh/ >> ~/.ssh/authorized_keys

You can try if it works by trying to connect without a password:

ssh $(hostname)

That’s it for the preconfiguration! In the next part of this blog series, we’ll use this host to run the appropriate Ansible scripts. Stay tuned for Part 2!

Rubix is a Red Hat Advanced Business Partner & AWS Standard Consulting Partner

Advanced MyST Usage: appendvar


Before we used MyST, I used to put some of the settings I needed in the script you get out of the box with Oracle Fusion Middleware 12.x products. One of the reasons for this, is that I needed Linux to resolve a $(date) command for my logfile name at boot time. With MyST however, it seemed that I could only add this argument as a startup arg in the weblogic console. The obvious downside here is that Linux cannot resolve the $(date) command this way. I thought this to be a MyST limitation, until someone showed me this trick.

This post is part of a Series in which Maarten Tijhof and I explore the inner and hidden parts of Rubicon Red MyST Studio which might come in handy at times.

Putting startup arguments in the can have certain advantages. Personally, I find it helpful to put generic startup arguments in the for a domain (such as java heap size), while setting server-specific startup arguments in the arguments field of the Weblogic Console.

There is a hidden feature to fill the file in your DOMAIN_HOME with certain values, and this trick is called appendvar. Like defining ‘users and groups‘, you can set these values using a specific key-value format in the global variables. First, you need to define the item you want added to the vanilla Out of the box, MyST already has an entry for disabling DERBY in 12c blueprints. You can add your entry to this comma seperated list.

patch.custom.appendvar.list (comma seperated list of custom setDomainEnv entries)


As you can see we added the ‘gclogs’ option to the existing patch.custom.appendvar.list.
Next you need to define the name for each entry and the values for those entries. Note that you can use Linux commands if needed, as this file gets parsed on Linux when the server starts.

patch.custom.appendvar.<item>.name (the name of the argument you wish to set)
patch.custom.appendvar.<item>.value (the value for that argument)

patch.custom.appendvar.gclogs.value=$EXTRA_JAVA_PROPERTIES -verbose:gc -Xloggc:$DOMAIN_HOME/servers/$SERVER_NAME/logs/gc_$(date +%Y%m%d_%H%M).log

You can also define for which servers these settings apply. All nodes will get the same setUserOverrides file, but MyST adds ‘if’ statements in the generated file, so it only applies the arguments to specific servers. By default, all servers (including the AdminServer) get the defined custom startup arguments.  You can change this targeting with the key-value lines below.

patch.custom.appendvar.<item>.target.cluster.list (accepts one or a list of cluster as input, useful if you only want the value applied to managed servers)

patch.custom.appendvar.<item>.target.server.list (alternative to target.cluster.list, uses comma separated list of servers on which you wish to target something)


The end result will look like this:
if [ "${SERVER_NAME}" = "AdminServer" ] ; then

if [ "${SERVER_NAME}" == "osb_server1" ] ; then
export EXTRA_JAVA_PROPERTIES="$EXTRA_JAVA_PROPERTIES -verbose:gc -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -Xloggc:$DOMAIN_HOME/servers/$SERVER_NAME/logs/gc_$(date +%Y%m%d_%H%M).log"

if [ "${SERVER_NAME}" == "osb_server2" ] ; then
export EXTRA_JAVA_PROPERTIES="$EXTRA_JAVA_PROPERTIES -verbose:gc -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -Xloggc:$DOMAIN_HOME/servers/$SERVER_NAME/logs/gc_$(date +%Y%m%d_%H%M).log"

By default, MyST will now add these entries to the nodes when provisioning a domain, as this is generated at the ‘patch-domain’ phase of MyST.

But what if you already have a domain and want to add these settings? Using an ‘update’ action will not push these values, as ‘patch-domain’ isn’t part of the default MyST update behaviour. You can however run a custom action to push these changes.

On the desired model overview, use the ‘control’ button, then choose ‘custom action’. As ‘action name’, type ‘patch-domain’ and press either enter or tab. Hitting ‘Execute’ will generate the for you on your Weblogic domains.


And that’s it! We hope you find this trick as useful as we did!


Hidden MyST Features, Part 2

Rubicon Red MyST can provision your Oracle Middelware environments, but it can potentially break existing in-use environments if you don’t know what you’re doing. This tradeoff exists in many provisioning tools, but it’s not necessarily something to be feared. In this post we’ll discuss the impact of various MyST actions, something that has confused many beginning users.

As many readers will know, MyST is very good in provisioning new weblogic-based Oracle platforms such as OSB, SOA and BPM on target nodes. To achieve this, MyST requires various powerful passwords.

First, you need to enter the password (or even better: a long SSH key) of the user you want to use on the OS level. Second, you need the SYS password of the database you want to use. MyST needs this to run the Oracle Repository Creation Utility (RCU) to add the proper schema’s to the database. Additionally, you have to set up the Weblogic password, as this is used to create the domain and to connect to the AdminServer every time you press the update button.

Although these passwords give MyST a lot of power to do the required steps, it can also be used to truncate existing database schema’s, or wipe entire Weblogic domains on the Linux Filesystem. The table below explains the actions available in MyST, the use case for each action and what the impact of each action is on the real-world Weblogic domain and MyST itself. But before we deep-dive in the use cases, we need to elaborate on models vs. instances.

MyST keeps track of the real-world environment by generating an ‘instance’ for each provisioned platform model, which can be a bit confusing. As a matter of fact, most users forget the ‘instance’ button in the MyST Studio interface altogether, as the ‘Models’ view almost shows the same thing. The instance is simply the latest provisioned model. This is important, because many users will expect MyST to always use the latest committed model.

Example: Imagine you provision an OSB DEV environment, which we will call OSBDEV[1.0.0][pr1][pm1]. If you apply a change and commit the model, the model will get version [1.0.0][pr1][pm2], but the provisioned instance will stay at [1.0.0][pr1][pm1] until you either reprovision (and fool MyST by hitting the ‘already provisioned’ checkbox) or press update. As you can see, a drift detection (which compares the instance to the real world) will give a different outcome then an update dry-run (which compares the last committed model to the real world).

Check for Drift

Impact Weblogic domain: No Impact
Impact MyST: No Impact

Use Case: Looking at differences between the real-world Weblogic environment, and the MyST Instance. A report is generated at the end of the action.

Note: MyST will connect to the Weblogic AdminServer and it will create a change lock, but after the drift detection it will cancel the change. If MyST detects an existing change lock by the Weblogic user, it will cancel the drift detection.


Impact Weblogic domain: No Impact
Impact MyST: No Impact

Use Case: looking at the actions MyST will do when someone presses the ‘update’ button. To achieve this, MyST compares the real-world Weblogic environment with the lastest MyST Model. A report is generated at the end of the action.

Note: MyST will connect to the AdminServer to create a changelock, but it will be cancelled when the action is completed.

Override State

Impact Weblogic domain: No Impact
Impact MyST: Impact (state only)

Use Case: In rare cases, your model will show an incorrect MyST state, in which you cannot perform certain actions. By overriding the state, you can let MyST think an environment is functioning properly, which allows for example the update action.

Reprovision (ALREADY PROVISIONED checkbox)

Impact Weblogic domain: No Impact
Impact MyST: Impact

Use Case: This action can be very useful, as it updates the MyST instance to the latest committed model version. This way, MyST thinks that this model version is already present in the real world. When you have an existing Weblogic environment, you can manually create a blueprint/model for this environment and use ‘Reprovision ALREADY REPROVISIONED’ to let MyST create an instance for this model. Next, you can use Drift detect to see if your model is correct.

Note: MyST will connect to the AdminServer host and verify that the aserver domain is present.


Impact Weblogic domain: Impact
Impact MyST: Impact

Use Case: This action is used most often and it pushes the MyST configuration changes from the latest committed model to the real world Weblogic environment. To do this, MyST uses the Weblogic username and password to connect to the AdminServer, it creates the change lock and applies all changes. Then the change is activated. If this fails, all changes in the weblogic console are rolled back.

To prevent this behaviour, first reprovision the model with ALREADY PROVISIONED on true. Next, use the ‘control’ button, then choose ‘custom action’. As ‘action name’, type ‘update’ and press either enter or tab. for ‘additional arguments’, use ‘-Ddo.not.rollback’.
Hit execute.

custom action

This will perform the update again, but when the session activate fails, MyST will simply disconnect, allowing you to login to the weblogic console and analyse any errors. In my experience, sometimes it helps to hit ‘save’ on a certain configuration item in the weblogic console, to see the root issue that is not shown either in MyST or the Weblogic log.
Note that this custom action applies all differences between the real-world environment and the instance, that is why you need the provision – already provisioned action.

Reset Drift

Impact Weblogic domain: Impact
Impact MyST: Impact

Use Case: This action will perform the same as an update.


Impact Weblogic domain: Impact
Impact MyST: Impact

Use Case: This action will re-install your entire environment! All existing schema’s will be removed and re-created. All domain configuration (expect for the oracle-home) on the target hosts will be removed. This action is useful for re-installing test environments that need to be clean. Never use this action in a mission-critical environment, unless you use the ‘already provisioned’ checkbox (see above).

Note: MyST runs the RCU to remove and create the schema’s required for the involved Oracle products. If you have other schema’s in the defined RCU database or other databases defined in datasources, these will not be affected.


Impact Weblogic domain: Impact
Impact MyST: Impact

Use Case: This action will remove the real-world environment, by removing all existing schema’s created by RCU and by removing all domain configuration (except for the oracle-home) from the target hosts.


Hidden MyST Features, Part 1


Provisioning and Maintaining your Oracle Fusion Middleware domains is pretty straigthforward when you’re using Rubicon Red MyST. The MyST Studio graphical user interface has gotten a lot better in the last couple of years, and is still actively developed today. In this blog series we’ll cover some Rubicon Red MyST features which are more advanced, and not always straight-forward. We’ll start easy, with some tricks.

Most of the readers looking into these tricks will know very well what MyST can offer and how it is beneficial. If however you’re new to MyST, let me briefly explain what it can do.

Rubicon Red MyST is a tool for provisioning and maintaining various Oracle Middleware products. It focusses mostly on Oracle OSB, SOA and BPM, but also supports other products, such as versions of BI Publisher, Webcenter Content and even simple weblogic or webtier installations. It uses the concept of blueprints and models, in which blueprints contain generic (environment-independent) settings and models inherit these settings to make them environment-specific.

Imagine for example an Oracle SOA installation in which DEV, TEST and PROD use the same listen port (which you set in the blueprint), but different SOA-INFRA database urls (which you set in the model).

As this is relatively easy to do, there are some more advanced tricks which you cannot find in the documentation.


If you set up a question mark as a value for a config item, MyST will remind you to set the actual value, when you are trying to run a reprovision or update of a specific model. The reprovision or update will fail and the log will display that the value needs to be set first before MyST can run. This can be usefull in various ways:

  1. Force passwords. You can set up a datasource in the blueprint, but as the password is always hidden, you won’t see that the password is not set or set incorrectly at the model level. By setting the password on a blueprint as a questionmark, you cannot roll out the model unless you overwrite that question mark with the actual password.
  2. Force endpoints. Although Database URL’s, SAF Remote Contexts and loadbalancers vary per model, you can set these values as a questionmark on the blueprint level, reminding the model-maker to set these values before continuing.

Another small trick: If you’re creating your own Global Variable and want to set a password, just use the word ‘password’ as part of the variable name. This way, MyST Studio will recognize that this is a sensitive value and after hitting ‘save’, your value will appear hidden.

Also, consider changing the db-sys-password value to some invalid text after the initial provisioning. This will prevent accidental termination of the database schema if you press the wrong button with a mission-critical environment.

Adding Users & Groups

Sometimes It’s necessary to add users and/or groups to the internal LDAP of weblogic. As of version 6.4 there is no section in the configuration screen dedicated to this. To add users, you have to get into the Global Variables section of your configuration.
If the user(s) you want are generic for all environments. You can do the following in the global variables of the Blueprint:

add.users (Comma separated list of users you want to add)


This section tells MyST you want to add users to the domain. Of course you need to specify these users:

USERNAME.password  (Password for USERNAME)  (optional group(s) you want to set for USERNAME)
USERNAME.description  (Description as shown in Weblogic)

user1.description=OPS user

The same can be applied for groups, in a similar fashion:

add.groups (Comma separated list of groups)
GROUP.description (Description as shown in Weblogic)

group1.description=OPS users
group2.description=DEV users

Using Global Variables in strategic places

MyST can be a bit repetitive for some parts, as you might need to repeat certain configuration items for each model over and over again. Imagine you have 2 datasources which go to the same database, but each use a different user. To set this up, you can configure the datasources at blueprint level but you would still need to set up URL and password at the model level. If you have 4 environments, you need to enter the database url field 8 times, and that’s just for 2 datasources!

One trick to save you some time is to create a custom global variable on the blueprint. You can name this variable any way you like, and set the value to a questionmark.


Next, hit save and copy the name of the variable you created


Within the same blueprint, now navigate to the two datasources. Because URL is usually not set on the blueprint level, you have to hit ‘show advanced properties’ on the top right to see it. In this field, paste the ${var.db.sales} variable field. As you hit the calculator in the top of the configuration panel, you’ll see that the resolved value now shows as the questionmark you’ve set earlier


This now means two things:

  1. At the model level, all you need to do is to overwrite the contents of global variable db.sales, and replace the questionmark with the specific database URL of that environment. This URL is then automatically used in all datasources which contain ${var.db.sales} in the URL field. You only need to set this URL once per model now.
  2. The questionmark which you set at the start works as a fail-safe. Provisioning or updating the model without overwriting the variable will throw an error, which prevents someone to provision a datasource with a non-existing database.

This concludes the MyST tricks for this round, we’ll post more advanced tricks in the future!


Advanced MyST Usage: EM Console tuning

Every Oracle Fusion Middleware Administrator wil recognise the complaint: EM Console is slow. The reason for this can differ, but it is most often caused by lots of composites which are deployed on your SOA-INFRA and the Admin Console collecting metrics on all of them. Oracle Support discusses this issue in Doc ID 1423893.1, in which they explain how to set various mbeans to cache certain EM metrics. Settings these mbeans has to be done through the cumbersome mbean tree. But what if I told you MyST can do it for you?

This post is part of a Series in which Maarten Tijhof and I explore the inner and hidden parts of Rubicon Red MyST Studio which might come in handy at times.

In Oracle Fusion Middleware SOA Suite, the Enterprise Manager is used to show composites, metrics and (if desired) even the composite instances with or without payload. Collecting all these values puts a lot of load on your AdminServer and it is not uncommon for the Enterprise Manager Console to take a very long time to log in. Especially when the amount of composites and the payload grows, logging in to EM can become quite a pain.

Oracle addresses this issue by giving you certain mbeans to tune the EM Console. Most of these mbeans concern caching certain displayed results so that the login becomes much faster after building up the cache. Unfortunately, the first user to log in is still stuck with the slow login time, as this time is used to build the actual cache.

Of course you can manually set the mbeans without MyST, just use Doc ID 1423893.1 and follow the steps to add the appropriate mbean values in the Mbean browser. You’ll be looking for the emoms.props bean. Once you found it, hit ‘properties’ and some default key-value pairs are shown.


As described in the Oracle Documentation, you’ll need to add some fixed key-value pairs to the list shown above. You can do this using ‘setProperty’ on the ‘Operations’ tab. Repeat the invocation for each of the four key-value pairs described in the document and you should be all set. Note that this has to be done for each EM Console you wish to tune, so you can imagine why this is cumbersome.

However if you have MyST, all you need to do is add these properties to the Global Variables of the Platform Blueprint. This way, you only have to set the desired tuning values once and they will be used on all models that inherit from that blueprint. Simply click the blueprint and hit ‘Edit Configuration’. In the Global Variable section, press ‘bulk edit’ to add the properties listed below. Hit ‘save’ or ‘save and commit’ on your blueprint to save your changes, you no longer have to save the bulk change seperately. The list below describes the properties you need and what they do:

--enable caching for FMW Discovery data--

--sets how long the cache is valid in millisec, below is 30 days--

--sets a timeout for a running discovery when a new user logs in

--skip certain metrics (and big SQL statements) so login is faster--

The tuning of the EM console is now added out-of-the-box for each new environment you provision based on this blueprint! But what about existing models?

First, select your model and run the ‘update’ command. Although you will not see any changes in the log, this changes your MyST instance so that it will use the new global variables in the future. In case you’re dealing with a 12c environment, you need to run a custom action on the same model and use ‘configure-soa’ as the action. Don’t forget to hit the ‘tab’ key, otherwise the command won’t stick. In 11g environments this step can be skipped because the ‘configure-soa’ command is part of the ‘update’ procedure.


Hit Execute and you should see the following in the MyST logs:


And you’re done! Note that these instructions closely resemble the method of setting mbeans on your MyST environments, as described by Maarten. However, setting mbeans has to be done on the ‘SOA Product’ section in MyST, whereas the EM Tuning parameters have to be added to ‘Global Variable’ section.

We hope these instructions will save you a lot of time, clicking though the mbean browser and especially logging into EM.

Optimizing the JVM for Oracle SOA Suite

This post is written for fellow SOA Suite Administrators, I will show how improper Composite cleanup in the Oracle SOA Suite will affect the performance of the Java Virtual Machines.

SOA environments are always in motion, as business requirements grow and developers keep adding services and support for frontend and backend systems. Depending on the size and complexity of the middleware environment, this can result in a constant flow of new composites, ready for deployment.

The Oracle Fusion Middleware SOA suite supports a nice way for service lifecycle management. It allows the SOA Administrators to have multiple revisions of one composite. If for example a new version of a composite is deployed, the revision number is increased and the new composite becomes the ‘default’ version. Older versions of the composite can still be used by older running instances. New instances will automatically use the new (default) composite. If for some reason the new composite contains a problem, an older revision can be set as the default.

Although this mechanism offers interesting lifecycle and fallback possibilities, it can become difficult for SOA Suite Administrators to keep track of all the composites which are deployed. After all, deploying new composites is always in the interest of the business, but the importance of proper composite cleanup is often underestimated or even forgotten. As we’ve seen with one of our customers, this can have serious impact on the performance of service calls and the throughput, because this is directly related to the way the Java Virtual Machine works.

The Java Virtual machine (JVM for short) is the underlying engine of every Weblogic server. To run all the compiled java code, it requires a block of memory inside the physical memory of the server. This block is called a heap and its size is often pre-set in the startup properties (using the Xmx and Xms arguments). The JVM uses about 25% of this heapspace as a nursery(1). New memory allocations will be pointed to this piece of the memory. As the need for memory grows, the space within the heap will run out. When this happens, the JVM will try to cleanup parts of its memory by performing a Garbage Collection. At first, the JVM will try to perform this process as quick as possible, minimizing the impact for applications running on the JVM. These attempts to clean memory quickly are called Young Collections and often don’t take more than a couple of milliseconds. In this step, the JVM checks if the allocated memory in the nursery is still used. If so, the memory is ‘promoted’ to the larger part of the heap. The rest is cleaned.

When however the heap can’t be cleaned with this ‘quick’ method, the JVM will try a more rigorous approach. A different kind of Garbage collection is started, which is called an Old collect (or Full Collect, depending on the JVM). In this process, the JVM will scan the entire heap to clear as much memory as possible: Marking and Sweeping to find removable objects and a compact stage to ‘defragment’ the heap. Because of these steps, the JVM will require a ‘stop of the world’ moment, which will pause all java activity. Depending on the size of the heap and the amount of cpu’s available, this can take up to several seconds. When the Old Collection procedure is complete, the remaining java code which cannot be cleaned, remains. This footprint is called the liveset.

As shown above, Young Collections are pretty harmless but Old collections can have a big impact on performance. To tune this process, the goal is to minimize the number and duration of Old Collections. One way to do this, is to reduce the liveset. After all, the Old Collection will go through the entire heap to find free space and a large liveset means that a high percentage of the heap

cannot be cleaned. Every Old Collection will try to scan this part of the heap, pausing all java activity, while nothing is gained. Additionally, the liveset also determines the ‘baseline’ on which new memory is allocated. If this baseline is higher, less memory is available until the JVM runs out again.

For example, imagine a heap of 1024MB, of which the liveset about 30%. This means that 300MB of the heap is allocated by memory which cannot be cleaned. When the JVM runs out of memory (e.g. 1000MB is used already), an Old Collection is started. This will scan though the entire 1000MB of memory and 700MB will be cleared. New allocations will start from the 300MB baseline.

Now Image this same heap with 50% liveset. This means 512MB of heap will never be cleaned after each Old Collect, even though the process will have to scan the same 1024MB. And because of the higher baseline, the JVM only has 488MB this time before it runs out again.

In a managed Weblogic server running SOA Suite, this liveset basicly exists of the soa-infra application and all deployed composites at any point in time. Because non-default composites are also loaded on startup, these will also remain in the liveset of the JVM the entire time. Reducing the amount of deployed composites on a managed server is an efficient way of reducing the liveset. In result, your Weblogic server will have fewer pauses and additionally, these pauses will be done quicker.

At our customer, an ANT script was build to undeploy all non-default composites. This way SOA Administrators can easily clean all the old composites before a new release is deployed. Within two weeks after the release is put in pace, the older non-default revision of composites are removed. This way a ‘composite backup’ is retained for a short period in which older running instances can still finish all steps. Also, by only maintaining a maximum of two revisions of each composite for a limited time, the liveset is reduced to a minimum. This procedure benefits all Java activity including the throughput of the load and resulted in a stable processtime with fewer outliers.

Grafiek blog Robert Jan liveset composites













From JRockit to HotSpot: Mission Control Templates

In this blogpost I’ll describe some of the difference between the JRockit and Hotspot JVM. Both JVM’s are Java Runtime Environments for running Java applications. Additionally you can see how some JVM Profile Recording options are still available in the new Mission Control tool.

Last year, we’ve upgraded some of the FMW environments at one of our customers from to 12.1.3. The project was pretty intensive given the size and importance of the environment, but after a while we pulled through. One of the things we’ve noticed is the impact that the new version has on the performance and the way we used to maintain our environment, specifically at the level of the Java Virtual Machine.

Continue reading “From JRockit to HotSpot: Mission Control Templates”