DevOps Zone is brought to you in partnership with:

Ranjib is a system administrator at Google. Prior to Google, Ranjib was a senior consultant with ThoughtWorks. He works on private cloud implementation strategies, cloud adoption, system automation etc. He has worked on both application development as well as system administration, for past 6 years. Prior to ThoughtWorks, Ranjib was working with Persistent Systems . Ranjib has done his gradation in lifescience and masters in Bioinformatics. Ranjib is a staunch FOSS supporter. Ranjib is a DZone MVB and is not an employee of DZone and has posted 13 posts at DZone. You can read more from them at their website. View Full User Profile

Infrastructure Tooling Patterns List

04.08.2012
| 11529 views |
  • submit to reddit

Following is my grouping of tools that I have learned/used as a sysadmin and DevOps dude at ThoughtWorks while maintaining our distributed infrastructure, setting up our private cloud installations, and in many different client gigs.

You can add some of these tools when required as your infrastructure/deployment/app grows. 

  1. Provisioner: Abstracts your vm / environment provisioning mechanism. Mostly relevant if you are on cloud infrastructure. Examples are boto and fog. Very important if you plan to do something like auto scaling. Gives you elastic infrastructure. 

  2. Configuration Management system: Lets you create reusable environments by expressing packages, services, files and other components via a DSL.  It also addresses cross platform issues. Puppet, Chef, Cfengine, and Salt are examples. A mature CMS setup will give you context aware infrastructure, like how your web server can automatically recognize the DB server, or how the load balancer can automatically recognize your web servers. A mature CMS setup will also incorporate the notion of environments and have versioned infrastructures like UAT, which can have app deployment version 1.3 and production version in 1.1 and staging in 2.0 etc. 

  3. Application deployers: Let you deploy your application. CM tools can do this too, but there are dedicated apps for this: Vlad the Deployer (Ruby), Capistrano (Ruby), Func: Fedora Unified Network Controller (Bash, Python) and Fabric (Python) come to mind. They also help you in creating ad hoc system automation. Most of these are SSH in a loop (or using GNU parallel). 

  4. Orchestrator: Functions similarly app deployers but incorporates middleware-like facilities like to do async command dispatching. Mcollective and Salt are examples. Both of them uses a middleware (Salt uses ZeroMQ while Mcollective can use any STOMP compliant AMQP) to broker 1->N, 1>1, N>M, N>N real time, and async dispatching. They can be used with platforms that dont have SSH and are massively scalable. 

  5. Monitoring solution: Keep tabs on performance.  There are 3 kinds of monitoring you'll need mostly.

    System (disk, CPU load, memory)
    Services (web server, DB server etc)
    App (I use a cuke script that checks how the whole app is working).

    A good monitoring solution is one which easily integrates with all other infra services, lets you define metrics (app response time, free memory, cached memory etc). It can also include customization notifications (email, jabber, sms etc) and escalations.  How the tools chart your metrics is also very important for understanding trends.  Reporting and event handlers are two important features here as well (use the event handlers in conjunction with the provisioner to get auto scaling features).

    Examples are Nagios, Zabbix, Zenoss and many many more. None of them are complete, but all of them can be complemented with some tool plugins (like for Graphite, an awesome charting tool). Nagios has text based configs but does not use any DB.  It's easy to install, scales well, and is mature hence integration with other apps is very easy. 

  6. Log management and log analytics: For tracking your logs.  Three parts again for this solution:

    Forwarding: a client that will sit in every VM and forward the log to a central location. Options are rsyslog, syslog, syslog-ng, Graylog agents, Logstash, Splunk forwarders etc.

    Gathering: A server that will accept all logs. Syslog-ng, rsyslog, Splunk, Graylog2)

    Analytics: In most cases you will be searching , indexing your logs for particular patterns. Graylog2, Logstash (both use ElasticSearch as the engine). Splunk (very powerful , very costly). A matured log management solution will let you set up alerts based on patterns (like failed transactions, 50Xs, 40Xs etc). 

     
  7. Supervisors: They observe a service and take appropriate action to bring them alive whenever they are down. Bypassing the whole network monitoring > event handler loop. Supervisors are very helpful for a shaky service. Monit, Bluepill, Godrb, etc. are some examples. A good supervisor has low memory/CPU footprint, provides fast healing capacity, and rich DSL for expressing a service state (like which port should be responsive, which process should be running, how to fix the process if it dies, or an alert when it takes an action etc). 

  8. Security, Hardening, and Auditing Tools: Specialized tools for strengthening system security.  Tools like Bastille ensures you have done the basic OS level hardening. It can also assess your infrastructure and lock it down if needed. Tools like PSAD and Snort uses IP tables' logs to automatically block intruders. Some of the CM tools like Puppet or Chef can be used to audit.


    Any other good additions to this list by category are welcome and encouragd in the comments!
Published at DZone with permission of Ranjib Dey, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Muhammad Faiz replied on Thu, 2012/04/12 - 11:50am

Really nice codification of the process in this post. I'd be interested in featuring this.

Jaffa Wify replied on Mon, 2012/05/14 - 4:38am

Infrastructure is basic physical and organizational structures needed for the operation of a society or enterprise, or the services and facilities necessary for an economy to function. Thanks. Regards, LookupExpert.com

Ragdu Bagdu replied on Mon, 2012/07/23 - 12:04pm in response to: Muhammad Faiz

Have a look to this Link for some great information on this product.

Bas Bihari replied on Tue, 2012/07/31 - 10:30am in response to: Muhammad Faiz

I like the article written on this site related to vans and i want everyone who wants to buy a van please visit SwissVantek for good deal.

Gas Bihari replied on Tue, 2012/08/07 - 8:53am in response to: Muhammad Faiz

Purchasing life insurance for the first time is challenging.But i will make it easy for everyone.See the link risikolebensversicherungtestsieger.de .

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.