Delivering UAT without any documentation

While we would never admit to still owning a 2nd generation iPad, the conference table was littered with every model of electronic device — phones, laptops, tablets. Our objective was simple: break the client’s rebuilt app… just as soon as we figured out how the old one worked so we could begin testing it.

The project started when our client came to us looking for UAT support. They were recreating and migrating an existing mobile application into their chosen CMS platform — Adobe Experience Manager. A move that was expedited as their current application hosting was expiring. Like many outsourced apps, our client didn’t own or have access to the development documentation or resources created by the vendor. Our client risked losing access to their product and content — a primary communications channel to their largest client base.

On the face of it the overall project was simple, recreate the mobile application to do exactly what the old one did — without the user base realising it has been recreated.

The challenge lay in testing the new application as none of the original specifications or manuals were available to inform the team of the business requirements or how the app should behave. This was a particularly unique situation. It’s not common for UAT to be undertaken by teams without product documentation as a reference!

We started by becoming experts in how the original app operated in order to start documenting the business requirements and developing scripts to perform various tests on the new app. Playing around with the original app gave us a real advantage and insight into how users’ interacted with the existing application.

“We spent time documenting known issues, key features and insights which were later referenced in the UAT process ensuring its fit for purpose.”

Once we had a good handle on the old app we began a full UAT on the new one using a range of tools and automated scripts we developed based on the old app. Rather than limiting our tests to functional and quality assurance measures that typically focused on ensuring the app worked — we purposefully tried to find issues that might make the audience disengage. For example — the existing in-app search feature was loading slowly. Technically it still works the same in both the old and new app — passing the functional test. But ultimately both fail the acceptance test. Users haven’t and wouldn’t want to use it.

We enlisted the support of our team across the globe to systematically undertake the UAT for geolocation bugs. We hooked up the mobile app through a proxy to see all the chatter between the mobile application and our clients’ server. As well as sharing various insights and issues we uncovered we found bugs in the original app requirements. This meant we could work with the developers to fix them for the new release.

While testing components structurally is one thing, UAT is an opportunity to go off script to see what we can do to break the app or system. We call this Edge Testing. There’s no better way to user-proof your application. Basically we ask our testers, ‘Can you break it in a way that the developer didn’t expect you to?’ We celebrated every time we found a bug — everything we could fix before launch was going to improve the user experience. We figured that if anyone was going to break it, it had better be us.

But for some improved user experience, the new (and upgraded) app looked exactly the same. This is what marked the success of this project: users didn’t even noticed the difference.

Tidbits and takeaways

  1. When using outsourced development vendors, ensure you know who owns the documentation and code (you, preferably).

  2. UAT is an element on its own in any development process, and should always be allocated due time. It should be used early in the development process, not at the tail-end rush to release.

  3. Most developers test products for functionality, but UAT is focused on fit for purpose. It’s difficult to test for every end user eventuality, but testing how to break (purposefully or accidentally) the application from the mindset of a customer should be core to producing a robust solution.

  4. Remember to think ahead for continuity and continual improvement. What will version 2.0 look like? What’s going to happen in the meantime?