Testing Cloud Services: How to Test SaaS, PaaS & IaaS

Posted on by Ben Rothke

Gartner Fellow and analyst David Mitchell Smith wrote in his report Hype Cycle for Cloud Computing last year - that while clearly maturing and beyond the peak of inflated expectations, cloud computing continues to be one of the most hyped subjects in IT.  The report is far from perfect, but it is accurate in the sense that while cloud computing is indeed ready for prime time, the hype with it ensures that too many firms will be using it with too much hype, and not enough reality and detailed requirements.

While there have been many books written about the various aspects of cloud computing, Testing Cloud Services: How to Test SaaS, PaaS & IaaS is the first that enables the reader to successfully make the transition from hype to actuality from a testing and scalability perspective.  For those that do not test the cloud solutions they are planning on implementing, they will rue the day when that same cloud solution will fail in production.

The book is an incredibly effective and valuable guide that details the risks that will arise when deploying cloud solutions.  More importantly, it provides details on how to test cloud services, to ensure that the proposed cloud service will work as described.

At 160 pages, the book is a great start to the topic. The 6 chapters detail a paradigm that cloud architects, managers, designers and anyone considering cloud services can use to ensure the success of their proposed cloud deployments.

The first two chapters are a very brief introduction to cloud computing.  The book expects the reader to have an understanding of the concepts of cloud computing and does not waste space with repetitive introduction to the topic.  In chapter 3, the authors detail the role of the test manager.  They write that the book is meant to give substance to the broadening role of the test manager within cloud computing.  They encourage firms to make sure the test manager is involved in all stages of cloud computing; from selection to implementation.  In fact, they write that it is only a matter of time until this service will be available in the cloud, in the form of TaaS – Testing as a Service.

Besides the great content, the book is valuable since it has many checklists and questions to ask.  One of the reasons cloud hype is so overly pervasive, is that the customers believe what the marketing people say, without asking enough questions.  It would have been an added benefit if these questions and checklists would be made available in softcopy to the reader. 

In chapter 4, the book details performance risks.  As to performance, an important aspect of selecting the correct cloud provider is scalability of the service.  This then requires a cloud specific test to determine if the scaling capacity (also known as elasticity) of the provider will work efficiently and effectively in practice.

A particularly important point the authors make is that when choosing a cloud service, many firms don’t immediately think of having a test environment, because the supplier will themselves test the service.  The absence of a test environment is a serious risk.

About 2/3 of the book is detailed in chapter 5 – Test Measures.  The chapter mostly details the test measures for SaaS, but also does address IaaS and PaaS testing.  The chapter spends a lot of time on the importance of performance testing.  

An important point detailed in the chapter is that of testing elasticity and manual scalability.  This is an important topic since testing elasticity is a new aspect of performances testing.  The objectives of elasticity tests are to determine if the performance of the service meets the requirements across the load spectrum and if the capacity is able to effective scale.  The chapter details various load tests to perform.

In the section on guarantees and SLAs, the authors make numerous excellent points, especially in reference to cloud providers that may guarantee very high availabilities, but often hide behind contract language.  They provide a number of good points to consider in regards to continuity guarantees, including determining what is meant exactly by up- and down-time; for example, is regular maintenance considered downtime or not.

Another key topic detailed is testing migration.  The authors write that when an organization is going to use a service for an existing business process, a migration process is necessary.  This includes the processes of going into the cloud, and backing the service out of the cloud.

With all of the good aspects to this book, a significant deficiency in it is that it lacks any mention of specific software testing tools to use.  Many times the authors write that “there are many tools, both open source and commercial, that can…” but fail to name a single tool.  The reader is left grasping at a straw knowing of the need to perform tests, but clueless as to what the best tools to use are.  Given the authors expertise in the topic, that lacking is significant.

The only other lacking in the book is in section 5.3 on testing security, the authors fail to mention any of the valuable resources on the topic from the Cloud Security Alliance.  Specifically the Cloud Controls Matrix (CCM) and Consensus Assessments Initiative (CAI) questionnaire.

With that, Testing Cloud Services: How to Test SaaS, PaaS & IaaS should be on the required reading list of everyone tasked with cloud computing.   This is the first book to deal with the critical aspect of testing as it related to cloud computing.  The ease of moving to the cloud obscures the hard reality of making a cloud solution work.  This book details the hard, cold realities of turning the potential of cloud computing, in the reality of a working solution. 

Had the designers of the Obamacare website taken into consideration the key elements of this book, it is certain that the debacle and PR nightmare that ensued would have been minimize and the administration would not have had to send out a desperate cry for help.  The Obamacare website will turn into the poster child of how to not to create a cloud solution.  Had they read Testing Cloud Services: How to Test SaaS, PaaS & IaaS, things would have been vastly different.













Kees Blokland Jeroen Mengerink Martin Pol  Rothke Rocky Nook 1937538389 978-1937538385


Ben Rothke

Senior Information Security Manager, Tapad

big data analytics cloud security virtualization, containerization & segmentation

Blogs posted to the RSAConference.com website are intended for educational purposes only and do not replace independent professional judgment. Statements of fact and opinions expressed are those of the blog author individually and, unless expressly stated to the contrary, are not the opinion or position of RSA Conference™, or any other co-sponsors. RSA Conference does not endorse or approve, and assumes no responsibility for, the content, accuracy or completeness of the information presented in this blog.

Share With Your Community

Related Blogs