Andrew Latham

A recurring issue in most computer system infrastructure is the System of Record (SOR)[1] which is a source of truth about the desired state or the current state of systems and networks. Many outsiders or senior management are left to believe that the planned state and the current state are the same but that rarely happens. I have written a few SORs and enabled others to be more accurate. Much of the work on modern container image manifests [2] has made great advancements to a more complete language to communicate the details of an endpoint. An issue I would love to resolve is the assumption that a single application is the SOR. The reality is that a data service is the ultimately the SOR and many things might read from or write to it. Today if I was asked to write another SOR I would setup RethinkDB [3] and establish a table named 'help' that pointed to a table named 'standards' and thus became self documenting system.

SORs should also be free to access offline in emergencies and distributed systems like RethinkDB and even Git are perfect for this type of setup. A support person or team can maintain a local copy of the SOR with little to no overhead. Even tools like MediaWiki[4] are great as they can be duplicated or distributed as a side effect of their design. For discovery there are established tools like DNS SRV [5] that are often overlooked. A popular solution for datacentres is NetBox [6] which is a fine step forward from RackTables [7] which served many for years and is still not a horrible solution. In my spare time I hope to integrate a DCIM [8] solution into Odoo [9] to connect the various organizational groups together.

Andrew Latham

My Personal Infrastructure

I have possibly too much physical hardware for one person to use. I like to setup laboratories and prove things out. I can spin up a VM on any given system I have in a matter of minutes. I care about my tools. Lets look at some concepts to creating a commercial and personal lab.

Commercial Laboratory

A commercial laboratory starts with product development and ends with product support in the lifecycle. The business will look at it as overhead when it falls under multiple budgets. Identify the laboratory as a shared resource.

  • Open Access Wiki
  • IPAM
  • Physical Hardware to match Production (trouble shoot hardware issue)
  • Software stack to match production (regression testing only)
  • API endpoints for testing
  • End user devices (tablet, phone, laptop) Apply roles and rights to the resource so that you capture the value. Example if a C-Level manager wants to do a private demo for a customer use the laboratory. Assign roles for network security, application security, network support, application support, customer support, product development, hardware support, production support and any number of roles. Do not call it a playground. Create environments within the laboratory for development, quality, production to enable the development and refinement of the product lifecycle and or the promotion life cycle.

Personal Laboratory

Limited resources are not a limitation in technology. A personal laboratory is not a business critical resource so you can build and destroy freely. You want to develop some simple processes for the build of VMs of various environments and make sure that it is easy. If it is easy for you to test something then you will test things with ease. Using a wiki you can also build up a complex environment without the resources others have by documentation.

  • Open Access Wiki
  • Laptop/Desktop you can afford to keep around
  • Network/NAS/Router that you can afford off of ebay

Test things

With decades of Open Source Software work I know to trust volunteer developed software more than commercial software. This is not tree hugging bias but actual experience. It is important to test things and build that experience for yourself. Proving out the impact of a change on a system in a laboratory vs production will obviously get you a raise some day so give it a go. Learn how to replicate a system package for package, config for config, and document the results of upgrades, changes etc.


With a personal laboratory feel free to test any software you read about, hear about, and or asked about. If you can setup random solutions in minutes and do it often then you will become confident in the process.

Andrew Latham

Playing with some libvirt stuffs and setup a quick task to get my HVM nodes working the way I want. Will update with some fine tuning over time.


- name: HVM Packages to install 
    name: "{{ item }}"
    state: latest
    - qemu-kvm
    - libvirt-clients 
    - libvirt-daemon-system

- name: Add user to group
    name: hvm
    groups: libvirt-qemu,libvirt
    append: yes
Andrew Latham

On systems like Debian Stretch with systemd the time sync is baked in but will not run if the legacy NTP package is installed at all so here is a howto/demo of what to do.

# apt-get purge ntp
# systemctl restart systemd-timesyncd.service
# systemctl status systemd-timesyncd.service 
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
   Active: active (running) since Fri 2017-08-11 10:09:11 CDT; 3s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 31413 (systemd-timesyn)
   Status: "Synchronized to time server ("
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/systemd-timesyncd.service
           └─31413 /lib/systemd/systemd-timesyncd

Aug 11 10:09:11 nodeone systemd[1]: Starting Network Time Synchronization...
Aug 11 10:09:11 nodeone systemd[1]: Started Network Time Synchronization.
Aug 11 10:09:11 nodeone systemd-timesyncd[31413]: Synchronized to time server (
Andrew Latham

Clean installs directly from the repos...

Using libvirt (virsh/virt-install/virt-manager) you can install from the HTTP repo of a Linux distribution to be super lazy. During installation you simply provide the URI in the location field or tools like virt-manager will have an option in the wizard.

For Debian you would use:

For CentOS which lacks a CDN mirror or I could not find it quickly, you can try:
Andrew Latham

My evil list of how to win!

  • No authentication required internally to read-only see the status and flow of a product - linkable steps any boss, support or manager can follow
  • Defined roles for gating products. Not named persons for gating.
  • Full or part time documentation curator. Great entry level position for person separate from the development or operations process.
  • Style Guides, Unit Testing, and Coverage reporting. Don't test everything, just most things!
  • Topic centric meetings with recorder (maybe the documentation curator)
  • Encourage failure disclosure, discourage boasting.
  • In project/agile user stories/etc use standard verb namespacing. Overly vague incomplete example:
    • Design widget requirements
    • Review widget requirements
    • Design widget solutions
    • Select widget solution
    • Define widget interfaces
    • Develop widget component A-Z
    • Validate widget meeting requirements
    • Define QA process
    • Define Release process
    • Release to QA
    • etc...
  • Document the product. Linking to the product source code is not a horrible idea to start documentation.
  • Iterate - work on bite sized items and keep moving forward while validating against the requirements at every step.
    • Validation is a process to mitigate project drift and accidentally report success. Do not reverse this idea.
  • Make it known that the functional deliverable is the product, reports that everything is awesome are not the product.
    • Management and or micromanagement is a distraction. Create clear communication that asymmetrical information is not useful for success.
  • Do not allow tools to decide workflow.
    • Tools can be replaces but workflow is hard to develop. Value the agreed workflow more than a tool
  • Morning pleasantries - Discuss life, food, sports, whatever prior to any issue, get a feeling for the day.
    • Call it morning pleasantries, really do...
    • A person's relative could have passed away over the night, never assume, be human
  • Standups have little value when roles are clear. Dispersed teams are impacted by standups
    • Decide if the roles are accurate and projects/jobs are defined.

TL;DR; Enable for humans.

Andrew Latham


Analyzing code to look for irregularities most commonly done with language specific Style Checks is a method of discovering things about the code and possibly some pointers to resolving issues.

Andrew Latham

Python the programming language and versions

Over the many decades Python has become a very stable language. From version to version more things are alike than different. In the last fire or more years people new to the industry have shown their cards by claiming it can't be done.

Not that hard

As I continue to hear sob stories about how hard everything is I keep reaching back to as you can see from the page the date and content show that it is not hard to support various versions if that is a goal.

Personal exp...

Almost exactly three years ago in 2014 I did a fast conversion of Ansible to work with Python 3 in PR 8322 which ended up in a land of personal preferences and overly complex solutions to simple problems. I have worked on converting other projects like PyPXE and Saltstack. In each case I hit walls where confusion is apparent and in every case I am supplying working tested code to prove it works.

Take away

Write better software, think about the future and the past. Work on problems like anything is possible because it is. Develop good clean interfaces and standard APIs in such a way that the software could be written in any version of any language and operate the same. Think before you code.

This rant was brought to you by Andrew Latham in a frustrated situation attempting to understand the lack of effort.