On November 11th, 2015 CloudBees and SOASTA co-presented a webinar about using Jenkins and CloudTest to achieve Continuous Performance Testing and Continuous Delivery. Our presenters were Brian Dawson, DevOps Evangelist from CloudBees and Tom Chavez, Product Marketing Manager from SOASTA.
You can view the recording and slides here .
Below are the answers to the questions they were asked during the session.
Q: When will SOASTA update their Jenkins plugin? The last update was Jul 20, 2014 which is now getting a bit long in the tooth.
A: We update the SOASTA plug-in as needed for continued compatibility and for new features. Is there a new feature you’d like to see in it? If so, please feel free to reach out to Tom (firstname.lastname@example.org) or visit the open source project on GitHub (https://github.com/jenkinsci/cloudtest-plugin . We have some new features coming out before end of year that will likely require an update so stay tuned!
Q. When doing these tests, are you depending on the live services you connect to? How did you coordinate that with your provider?
A: For a CloudTest load test, you can direct to pre-prod or staging wtihin your company so there’s no external services hit, but for a large load test in production, you can either filter out the requests from third parties (not recommended) or work with them and inform them that you’re doing a production load test at a certain volume. CDNs work with you on this and other third parties, such as API partners, should support this, as this level of load will come during customer peak periods, too, so better to catch issues in off-hours before they have real customer impact.
If you are already (or just suspect that you may be) having issues with third party services, you can add mPulse (http://soasta.com/mpulse ) to your site / mobile app to monitor the performance not only of your app but also any third-party content your app uses, even your own content served from your CDNs. By monitoring every user interaction, you will know as soon as your customer does if they are experiencing performance issues, even those that aren’t coming from the servers you directly manage.
Q: Does CloudTest allow SOA testing i.e. testing web services with SOAP over HTTP/JMS?
A: Yes, CloudTest supports more than just web http / REST traffic: also direct-to-database, websocket and SOAP.
Q: How does the SOASTA "visual" test design relate/compare to something like Selenium?
A: Selenium tests can be authored in a high-level language or with some recorders and typically execute in browsers, which limits their viability for load testing at very high levels, e.g. more than 1M virtual users. Selenium is great for functional testing whereas CloudTest is great for creating maximum load and measuring the performance of the app/site, along with some level of functional verification of results to catch app errors at scale, too. Selenium tests could be executed in parallel with a load test for deep functional testing of a site under load at scale.
Q: What if a SOASTA composition needs to launch a grid to run in the CI pipeline?
A: The ability to spin up load generators in the third-party cloud space is coming soon in a future release. Today our customers are using internal load generators, manually launching grids before the tests, or have made their own automation scripts to start up a grid.
Q: What's that DEV@cloud?
DEV@cloud is a fully-managed, hosted Jenkins service . Providing an instance on Jenkins Ecosystem (CloudBees Jenkins Platform, SCM repositories, ALM integrations etc), including the ability to view billing and cloud usage information, manage on-premise license keys and manage users across multiple accounts. Sign-up at: https://cloudbees.com/jenkins-cloud-signup
Q: Is CloudTest licensed? What is the cost?
A: CloudTest is a commercial product, starting with a free Lite version that allows unlimited tests at up to 100 virtual users. Pricing above 100 VUs is available from SOASTA at http://soasta.com/cloudtest
Q: So, out of curiosity, is there any step-by-step information or something similar (walkthroughs, maybe), from CloudBees to help us out (that are still learning setting up Jenkins, Jenkins + Docker, etc)?
Absolutely, in addition to the community information in places like:
CloudBees provide a wealth of additional resources to help with FOSS and CloudBees Jenkins deployment:
Tutorials e.g.: Jenkins Workflow
And you can always reach out to the CloudBees team for help at email@example.com
Q: I have frequent changes coming in the application where I am unable to do the continuous testing since I have to modify the scripts frequently. How do I handle such a case? Will Jenkins be useful in this scenario?
[Brian]: I believe your situation is such that application functionality is changing at such a rate that you can’t create test scripts fast enough to ensure optimal functional test coverage (as opposed to unit or non-functional tests.)
A couple phrases come to mind when I hear this:
Don’t Let Perfect Be the Enemy of Good
Right-size your test
Leverage hybrid or progress level-of-detail tests
Implement a test-centric strategy such as TDD/BDD
In more detail, some possible approaches are:
Don’t Let Perfect Be the Enemy of Good. Even if you don’t have complete test coverage, you should still be doing continuous testing. In fact, achieving comprehensive coverage for legacy code /or maintaining it for fast changing applications is an ongoing struggle. But start where you can, then improve your process and coverage gradually over time.
Right-Size Your Test. Consistent with agile principles or eliminating waste; don’t try to test everything. Instead, identify key areas of application functionality and create tests for those. Once tests are created they become potential maintenance and pipeline performance overhead. Focus on tests that address value.
Leverage a Hybrid Testing Approach. If you are not continuously delivering to production, then create quick, strategically-chosen tests to support continuously testing as part of CI, within your iterations. Then target completion of more comprehensive scripts in time for your releases.
Ideally Implement a “test-centric” strategy such as “TDD/BDD” requiring automated tests be created for each feature prior to implementation of the feature. This ensures that at least every new feature has automated test coverage. This is a more difficult transformation but will pay dividends in the long run.
[Tom]: If your new features are coming too quickly, you might need more QA engineering resources or commitment to QA, but I’ll leave that to your management. You could focus on automating your regression tests so that the unchanging parts of the app are still tested automatically, to ensure that the new features are not hurting functionality or performance of the existing features.
Q: Regarding the gradually incremental performance tests: What kind of real performance bottlenecks do you expect to find during unit test stage?
A: At unit test stage, you’re looking to set and then continue to test against a baseline for performance, to make sure that performance starts good and stays good or gets better. If the new code doesn’t run well with low load, then it likely won’t run much better at higher load, even on bigger hardware. Even in the initial stage, you can increase the load to learn where the bottlenecks are, whether it be CPU, memory, network, database, etc.