Breaking Things Productively

Have never used Chaos engineering, but the idea is interesting. Second read about the concept:

How to Use Chaos Engineering to Break Things Productively  by  Sam Bocetta in Infoq

More people connected to more servers, increased reliance on complex distributed networks, and a proliferation of apps in development mean more opportunities for data leaks and breaches.

Modern problems require modern solutions, as Amazon found out the hard way. Netflix escaped with minor inconvenience by being prepared.

What did they do differently?

Amazon Web Services (AWS), Amazon's cloud-based platform, experienced an outage on September 20, 2015, that crashed their servers for several hours and affected many vendors. Netflix experienced the issue as a blip because they've been there and done that when they changed their service delivery model. This led their engineering team to craft a unique solution for software production testing.

The solution? Chaos as a preventative for calamity. It's predicated on the idea of failure as the rule rather than the exception, and it led to the development of the first dedicated chaos engineering tools. Otherwise known as the Simian Army, they're called Chaos Monkey, Chaos Kong, and the newest member of the family, Chaos Automation Platform (ChAP).

What Are the Benefits of Chaos Engineering in DevOps?

Focusing only on a network environment and the associated security considerations (because the world of chaos engineering is quite large), we have already seen it as a positive force in an already strong cybersecurity market for improving business risk mitigation, fostering customer confidence, and reducing the workload for IT teams. If you're a business owner, you'll be blessed with happier engineers, reduced risk of revenue loss, and lower maintenance costs.

Customers, whether B2B or B2C, will enjoy greater service availability that's more reliable and less prone to disruptions. Tech teams will be able to reduce failure incidents and gain deeper insight into how their apps work. It will also lead to better design, faster mean time in response to SEVs, and fewer repeat incidences. ....  " 

OnTop-Down Testing

Simple but well stated.   How do you test?   Whats important?  Useful tips:   See also 'Chaos Engineering', at the tag below, now also in common use.

4 keys to a top-down testing strategy
By  Hans Buwalda, Chief Technology Officer, LogiGear

It's difficult enough to keep up with advances in software testing. Add to that the growing number of misunderstandings between test teams and leaders about how, where, and when to test, and it becomes even more challenging to achieve on-time, quality releases.

If you involve senior leaders in the testing process from the beginning, you can overcome many of these problems. While those leaders won't be doing the manual work, they should understand at a basic level what it takes to test and deliver great software.

From there, it's possible to establish leadership from the top by setting clear expectations around quality and by providing the proper support for team members. This support includes education, training, tools, and more.

As firms look to adopt testing in continuous delivery, they are questioning whether test plans still have a place. They do. Modern test plans can provide a vehicle for creating and communicating the test strategy and approach. They give teams an opportunity to communicate assumptions and approaches, including exclusions such as why a factor wasn't tested.

Here are some key testing tips to keep in mind : ....

Chaos Engineering at Linkedin

Have only interacted with 'Chaos engineering' once.   In particular to look at the broader idea of maintaining process models.  Here a technical view of it used with a popular service. 

Chaos Engineering at LinkedIn: The “LinkedOut” Failure Injection Testing FrameworkLike  | by Daniel Bryant in Infoq

The LinkedIn Engineering team have recently discussed their "LinkedOut" failure injection testing framework in more detail. This framework supports the generation of hypotheses about application and service resilience, and enables failure to be injected to a specific request via the LinkedIn LiX A/B testing framework or via data in a cookie. Failure scenarios that can be tested include errors, delays and timeouts. The LinkedOut project is part of the larger "Waterbear" initiative to encourage every team at LinkedIn to contribute to resilience engineering efforts.   ... " 

Evolution of Chaos

Excellent piece.   Have only rarely see this concept used in the enterprise, deserves more examination.    Here with an intro and then pointers to more.  My tag points to more resources I have gathered, including a free pdf book ....

The evolution of chaos
Kolton Andrus explores the evolution of chaos engineering and explains why it’s becoming the go-to approach for building resilient systems. 

This is a keynote highlight from the O'Reilly Velocity Conference in London 2017. Watch the full version of this keynote on Safari.

You can also see other highlights from the event. ..... " 

Chaos Engineering in Practice

Don't remember ever hearing of Chaos Engineering, but had done some engineering in Chaotic, non predictable situations.  It should be noted that this is not about chaos theory, in a mathematical sense, but rather testing and adjusting complex systems.   Nora Jones in InfoQ describes it in practice.

Free 81 page Book on the topic via O'Reilly,

Where they describe it and its development and use by Netflix:

Building Confidence in System Behavior Through Experiments.

" .... With so many interacting components, the number of things that can go wrong in a distributed system is enormous. You’ll never be able to prevent all possible failure modes, but you can identify many of the weaknesses in your system before they’re triggered by these events. This report introduces you to Chaos Engineering, a method of experimenting on infrastructure that lets you expose weaknesses before they become a real problem.

Members of the Netflix team that developed Chaos Engineering explain how to apply these principles to your own system. By introducing controlled experiments, you’ll learn how emergent behavior from component interactions can cause your system to drift into an unsafe, chaotic state.  .... "

How might this be integrated with forms of process modeling.  like BPM?  Could the testing be applied to a process model?