I recently volunteered at the awesome Ministry of Testing’s Testbash UK 2022. A huge draw for me was meeting the wonderfully talented Julia Pottinger who, with the backstage support of her QA partner Orane Findley wowed the attendees with her characteristically clear and simple explanation of “Next Level API Automation”.
Julia’s talk included the following slide, which detailed 6 important high level scenarios to consider when performing API tests. I thought it might be good to show examples of how, you can execute tests in all 6 areas using Postman Flows.
Julia Pottinger’s API Checklist. Numbers added by me relate to this blog post.
Scenario #1 Schema Testing
As JSON parsing seems to be done on a field by field basis in Flows, it doesn’t currently seem possible to extract the entire schema from the response body in order to compare it with a stored value (correct me if I’m wrong folks!). However, if you want a visualisation that your schema test stored within the tests tab of your request has worked, then here is a good way to do that.
Scenario #2 Security Testing
You can use the create data block to add different data sets to run against a request. I walk through a very aesthetically pleasing example here. 😻
Scenario #3: Workflow Testing
As highlighted by Julia Pottinger at the Ministry of Testing’s amazing Testbash UK 2022, testing an end to end workflow through API calls is often essential. Think about it logically, how do you know if a delete call has worked if you don’t then check the data isn’t then available? Using Flows to test workflows is a major use case for this Postman feature, because you can see exactly what’s happening. Here is an example workflow test using Postman Flows.
YouTube video showing how to perform a typical workflow test using Postman Flows
Scenario #4: Negative Testing
Here I cover 2 different but connected scenarios:- 1. You want to verify the status code and message (e.g. 400 Bad Request) 2. You want to verify a string within the error response (e.g. “the length must be 3-18 characters”
An added bonus is generating test data within Flows using both lists and records, which although might not be ideal for large test data sets, would be very handy for smaller ones such as testing what happens if 3 different invalid inputs are entered for the same request.
YouTube Video showing negative testing using Postman Flows (response codes, status and response body message check)
Scenario #5: Performance Testing
So we know that scaled and in-depth performance testing is one of the few things that Postman isn’t really built for. Although Flows is no exception to that, if its cheap and cheerful performance testing you’re after (say, executing the same scenario 10 times and checking the response time is under a certain threshold) then that is absolutely doable.
Scenario #6: Positive Testing
Last but certainly not least – time for the happy path! 🙌
In this example, I show how it is possible to execute 6 endpoints at the same time, and apply the same conditional logic to all of them to check:- * status code is 200 * response body value is correct (“status: UP”)
This gives a really clear indicator when running a health check against several different API’s at the same time. Which is one of the huge benefits of using Flows.
YouTube video showing happy path testing using Postman Flows
And that’s it! I hope that structuring this post around Julia’s essential test scenario’s made sense to you, and that you were inspired to try Postman Flows next time you are looking to test your API.
However, I think when we’re learning, it’s good to have examples we can refer to – that’s always the shortcut I use to get inspired anyway! So here are some working examples I created to help get those creative juices flowing. Note these are all basic Flows that may form part of a larger Flow should you wish to do something more advanced.
All of my Flows use the Restful Booker API, written and maintained by the amazing Mark Winteringham. Click on the links below to access a video and short tutorial on each Flow.
In this instance, we create a new booking, and pass the created booking ID to a request to GET the information about that ID. This is done via the the “assign values to variables” link within a request without the need to use the now deprecated Assign Variables/Create Variables blocks.
Video showing use of “Assign values to variables” link to define and pass variables – remember to create an example first!
Flow #2: Looping data
*Updated May 2022* If you want to loop through a response and apply a rule to it then you can find an example picture below. Here I take the ID’s from a GET all request, loop through them using a FOR EACH loop and for each of the ID’s in the request I output to a terminal if the booking id is greater than 100.
For Each Loop Highlighted in Green
Another way to do this is via the Create Data/Loop Over List Flow, also shown in the picture above. Using these blocks, I also execute a further request using the ID from the loop e.g. to GET information on the booking. I then output the test run results to a Test Summary block.
The steps are as follows:-
Ensure you have an example saved in your collection for the requests you wish to use in the Flow
Add a Send Request block
Add a Create Data block, using the data input to select a value (in this case /data/body/ which is a list of booking ID’s)
add a Loop Data Until block, which tells the Flow you want to do something with the data you’ve created
Add further blocks to do that thing e.g. Send Request (adding the variable of bookingid you’ve generated above) or conditional logic
(optional) to convert the data back to its original state, add an End Loop Block
Add Terminal/Test Summary Blocks to output the data so you can ensure it works – it will dynamically update with new information each time the data passes through the loop
When Looping, you must first Select the data you wish to loop through
Flow #3: Take Inputs From Multiple APIs
This example shows how you can take requests running against two different environments from two different collections and perform a Check on them both to output to a terminal.
Video showing outputs from two API requests flow into the same check block in Postman Flows
Flow #4: Passing Data Between Requests
From version 9.5.0, you can use the Create Durables block to complete the task of passing data between requests – as long as you have created an example API request first! By adding the token as a durable type, the generated value can be persisted throughout the Flow.
Video showing data being passed through a Postman Flow using Data Durables – Remember to create an example first!
Flow #5: Generating a Test Summary Report
My fave Flow! Takes tests run as part of 3 different requests and outputs the results to a single Test Summary Block. Note these can be from entirely different collections if, for example, you want to generate a quick eyeball test report from across all your critical APIs.
I’ve been wrapping my head around the latest Postman Flows early access release, and this one is a biggie. Of course it is still subject to further change still, but as outlined by the team in the linked post, the major changes to the current beta version of the low code API workflow feature called Flows are:-
Massively simplified list of blocks, probably around 50% are left (those that didn’t make the cut include Test Summary, Create Durables and Conditions blocks to name a few)
Youtube video showing the simplified block list in Postman Flows
Back by popular demand, the return of the Start button – don’t call it a comeback!
End of Terminals – log blocks can now be added instead, which pumps data to the console log
YouTube video showing console log entries appearing for tests following execution of a Flow
Webhook blocks – can now be added in order to trigger Flows from the Cloud – this will make CI implementation of a Flow possible as the Webhook URL that is generated when a Flow is created can be saved and called called like any other Postman request, as well as allowing a Flow to be triggered automatically by an event, say, a Slack or Discord message
The arrival of Flows Query Language (FQL). FQL aims to low code-ify data that is used in our API requests and responses, to allow that data to be easily queried, accessed, reused and changed in a much simpler way than by writing complex JavaScript pre-post scripts against the APIs themselves.
I suspect FQL will be subject to tweaks, but everyone acknowledges that the click and hope method used prior to this was pretty painful. To my mind it looks a lot more technical (and therefore a bit more scary for a new user to pick up) this way, so I’m hopeful that with plenty of feedback the team can continue to make the experience simpler. However FQL allows us to:-
Generate standalone data (e.g. current date/timestamp) to use
Pull data from a Flow (e.g. a response body value) to use
Create our own variables (e.g. no times to iterate a test) to use
From what I can gather, any of these can get fed as an input into an evaluate block which can query that data, then output the result. So for example, GET all bookings, then evaluate which booking ID’s are greater than 7, then output those booking ID’s which meet that criteria into the Flow to do what you want with e.g. add an if statement. I can’t quite get this to work at the moment though, so I’ll intend to update this blog once its a bit clearer in my mind!
Youtube video shows an early attempt to work out evaluate block using FQL
In this series of blog posts, I give short tutorials on how to accomplish something using Postman’s no/low code API feature Flows.
This time it is generating test data. If you need to run a set of steps repeatedly to create data before you can execute your tests, then here’s an easy way to do this, no code required.
Steps: 1. Create a flow of the steps needed to generate data e.g. Send Request then x,y,z. 2. Add a Loop N Times Block, with an inbound connector of Number. Select how many iterations you wish to run in the number block (e.g. I want to create 5 bookings, so will enter 5) 3. After the loop N Times block, add a Create Data block. Inside this block, create a list pointing to /data (in other words, the numeric input to the block that you have created above) 4. Add a For Each block after the Create Data block. This will connect to your steps created in step 1 above, and tells flows that you want to iterate through the looped data. Ensure the For Each block is set to pick up /for 5. Run the Flow and check the data has been correctly generated, hurrah!
Here is a short video running through these in more detail.
Youtube video on how to generate test data using Postman Flows
As discussed by Postman, Postman Flows has two different types connecting one block to another:-
Connection – the solid line
Signal – the dotted line
We use connections as standard. But it is useful to know how and when to use signals. Signals change the default order of execution, so if you want to make sure that block B waits for block A to complete before kicking off, then use a signal to do this.
How To Add A Signal Connector
Click the grey dot in the bottom left hand corner of a block e.g. block A
Drag the block to the “On” box of the block you wish to pause e.g. block B
You should see a dotted line connect the two, and the On change to Off in block B
The Off indicator is telling us that block B will be considered switched off until block A has executed. During execution of the Flow, as block B executes its status indicator will change to On.
Tutorial Video
Youtube video demonstrating the two kinds of connectors
I hope you’ve enjoyed this tutorial, and weren’t too distracted by my pesky moggy. 😺
Here’s a short tutorial post showing how you can fire off a message to your Slack channel when using Postman’s low/no code feature Flows. The good news is this is really easy, and you don’t need to know how to code to do this.
3. select the workspace you want to post a message to
4. On the newly created app, select incoming Webhooks on left hand menu
5. Toggle Activation to ON
6. Click Add New Webhook to Workspace
7. Select Channel to post messages to. Copy cURL request.
8. Open Postman, click Import button
9. In Raw tab, paste the cURL request from Slack copied in step 7
10. Save request in a new collection (e.g. “Slack API)
11. Create a dynamic variable for request body message text AND webhook token (optional). Save variables in new Slack API environment.
12. Send the POST request. Confirm Slack message received (yay!) Save example in Postman.
13. Open your Flow. You can now add a Send Request Block for the POST Slack Message request, adding in a variable for the message you wish to send. Remember to add to the reject output of a conditional block if you only want to send if something has failed.
Et voila! Your message will now be posted into Slack should the condition fail to be met.
Or check out my video to see this working:-
YouTube Video Entitled Postman Flows Send Message To Slack
TL:DR – see below section for a list of useful resources to begin learning more about Application Programming Interface (API) testing.
Today I got a message from Coders Guild software testing graduate Alexander Olakunle, who I have to say I found to be a very dedicated and driven person, who has worked incredibly hard on the course and landed his first testing job with Ordinance Survey (congrats again Alexander!). He asks:-
During the short training course, we cover API testing, but only from a high, theoretical level.
I thought it might be helpful to others (that means you!) to list some excellent, beginner-friendly resources to try out if you’re in a similar position to Alexander.
Talks
If you like seeing how different people tackle the same problem, I’d definitely recommend taking a look at the Test.Bash(online) talks from the Ministry of Testing. 4 different sets of people answered the same challenge, and gave demo’s on how they did it – so many useful tidbits of practical insight there.
Dave Westerveld has created a great book (published by Pakt) called “API Testing and Development with Postman” which serves as a fantastic introductory book for API testing (don’t let the development bit of the title put you off!).
If you complete that course, you can then go on to do my Test Automation Course API Test Automation with Postman, particularly if you’ll be using the Postman tool for your testing efforts. If REST Assured is something you’ll be using, Bas Djikstra has a TAU course on that too.
Certifications
I’ve only very recently become aware of this, but there is an organisation called API United, who have created an API testing certificate. It currently costs £170 to take the exam in the UK, but the syllabus is available online so its possible that this would be all it costs you to get the API-U Certified API, Rest & Microservices Tester (CARMT) with Postman or Karate
Screenshot showing the certification title and API United logo
Of course certification isn’t for everyone, but if you feel it would benefit you it might be worth taking a look.
More Advanced Stuff
Lewis Prescott – A master of all things contract testing, anything he appears in is a great resource if this is an area of API testing you want to learn more about, especially his podcast series “How to start API Contract Testing“
Mark Winteringham – Not a lot of people know this 😉 but Mark is gearing up to publish the final version of his incredible book Testing Web APIs. Its got loads of practical exercises in, so if you enjoy working through those you will get so much out of reading this book.
Of course, its well worth doing your own research, and I’d love to hear if you have found other resources elsewhere that you’ve benefitted from – drop me a comment and let me know, that way I could learn something new too!
For the last two months, I have been working as interim QA Relations at Mailinator. It isn’t often in our careers that we’re able to have a risk free go at trying something new. But wanting a bit of a gap from leaving my full time testing role to starting my next one, this opportunity was one I had to grasp.
Would I like to do QA/Dev Rel type stuff in future? Was it what I expected it to be? Would I recommend it to others? Read on to find out my honest opinions…
What did QA Rel look like?
Despite being almost 20 years old, Mailinator had never had a QA Rel before. Therefore, this was a great opportunity to evolve and learn from each other as we went along. Alongside weekly meetings with both the wider company and individual 1:1’s with the CEO, I was given a lot of free reign and some loose guidance along the following lines:-
Get access to social logins and regularly post interesting stuff
Learn more about the product itself
Write a blog post or two for the site
Give your professional insight as a tester on how to improve the service
Having a team that were enthusiastic and super keen to hear the views of a tester (their key customer) gave me confidence to give my opinion on a lot more areas than I first thought I would. The following things give a flavour of the key activities I got involved in:-
Social Media Stuff
What I did
Tweeted and Posted on LinkedIn and Twitter most days of the week
Polled users to find out what freebies they prefer
Created 6 tutorial video’s for an Essential Guide blog post
Penned 4 further blog posts
Learned how to use Yoast to improve a posts Readability and SEO scoring
screenshot of Mailinators blog page – all blogs shown were penned by myself
What I learned
Creating a blog for a company is a different ballgame to creating one for yourself. I thought this would be the easiest part of the role, but it was one of the hardest. I learned to give myself plenty of time both to write the initial draft of the blog, improve the copy using the plugin Yoast to get good SEO, accessibility and readability scores. I also took the extra time to insist on getting all posts reviewed by the team. The feedback definitely helped make the blogs better quality and avoid a few clangers!
Also – people really do not like to follow companies on Twitter and LinkedIn. I found it much harder to get “engagement” through the Mailinator account than when I posted as myself.
Testing Stuff
What I did
Created a Trello board of new bugs, customer insight and recommendations
This wasn’t part of the brief, but it is actually impossible to spot something that could be better and not feel compelled to be a part of making that happen. My Trello board was shared with the team, and included a lot of annotated screenshots (I use Techsmith’s Snagit – a paid tool but worth every penny).
What I learned
Not everybody loves Trello! There were a few bugs that made their way through the system and were reported by users that I had on the Trello board beforehand. I guess the lesson there is to find better ways to highlight important bugs and find a way to communicate that works for the whole team.
Reviewing Existing, Revealing New Features
What I did
Created Mailinator’s first public API in Postman, including a “run in Postman” button to easily get up and running.
Tweet from Kin Lane, API Evangelist, praising the Postman Mailinator Public API Workspace
Gave insight and feedback on a new pricing strategy and other features
Completed a competitor analysis
Created a document identifying 10 ways the company could improve that I debriefed with the CEO
What I learned
This was mostly stuff I hadn’t done before, so trying out new things and getting positive feedback was really rewarding. Sometimes its worth pushing yourself – as long as you’re not BS-ing about your experience beforehand, everything is a learning curve!
Also, at my leaving meeting the lovely CEO Paul said that having someone on board the team that was so enthusiastic about the product being a sounding board for new features meant that there was stuff now baked in to Mailinator that otherwise wouldn’t have been. That’s something I didn’t see coming, but a nice impact.
Also, I love Postman. Tell me something I don’t know. 😉
Working with 3rd Parties
What I did
Lined up guest bloggers to continue to provide content in the coming months
Coffee with Simon Tomes, community boss at MoT, to talk all things community and ask advice
Asked the community for a recommendation and brought on an amazing Tech Writer to improve documentation
Liaised with Ministry of Testing Marketing, Sales and Community peeps to understand how they could help Mailinator to improve their interactions with the testing community
Arranged for Mailinator to Sponsor a MoT event
Liaised with the community and Postman DevRel Sean to further improve the public API collection
Worked with the lovely Himanshu at cloud based cross browser testing provider LambdaTest on exploring a possible integration. This involved investigating LambdaTest’s capabilities, asking others in the community what they thought of LambdaTest, and doing a proof of concept to understand how any integration would work. Have to say I was pleasantly surprised by what this platform could do and how positive others were about it – so much so that I used it to do a tech test when applying for a new role.
What I learned
I learned that speaking to other people and forming genuine and positive relationships can be done, and was much easier for me than I thought it would be. (trumpet well and truly blown there Beth!) I made sure at the start of the assignment with Mailinator that QA Rel was not going to be another word for Sales – an area which I’d be terrible at and which they 100% agreed with – and I definitely think that is an important principle to stick to.
At least 2 of the above 3rd parties have reached out to me to ask if I’d be interested in doing further work for their organisation, so this feels like something I did a pretty good job at.
Metrics
What did I do?
Getting before and after data on our social media activity wasn’t part of the role, but I felt like it should be. If for no other reason than to measure my own success.
Increased number of Twitter Followers by 112%
Increased number of LinkedIn Followers by 554%
Increased visitor metrics to Linkedin page by up to 850% (average 200%)
We aren’t talking about huge numbers here – for example we crossed the 1000 followers mark on Twitter a few days before I left. But given Mailinator had been on Twitter for 12 years, I think this was a pretty good milestone!
I learned that all of this stuff isn’t really about numbers. I’d sooner have one meaningful conversation with someone than 100 likes to a post. But, over time, they do give trends and they do tell a story. Focus on the macro side of things.
Would you recommend QA/Dev Rel?
In much the same way that since home schooling last year has made me appreciate the teaching profession in a whole new light, I now feel the same way about Developer / QA Relations.
Dipping my toe into this stuff for, lets face it, a very short amount of time, has really made me realise what bloody hard work this stuff is! There is a world of difference between doing this stuff for yourself, and being paid to do it for someone else. It takes a lot more energy than you’d think, even for small stuff like writing a tweet in just the right way.
I think I enjoy talking about products, tools and companies that I genuinely am already passionate about – and Mailinator 100% fitted that category. This made things so much easier, because I didn’t have to fake authenticity.
Would I recommend it? I think perhaps as a side hustle its a fun thing to do. And I’ve met some mega people that I’ll definitely be staying in touch with and hopefully will help out in future. I hope for myself and other testers, I have done a good job at persuading the Mailinator team of the importance of decent swag. But it absolutely made me realise that I missed testing too much, and after working in QA for 15 years, that’s definitely where I want to be. I’m so lucky I had the chance to try this out, and am so grateful to the incredible team at Mailinator for helping make it such a blast.
I start my next adventure tomorrow, with Ada Health, as a Senior Quality Engineer.
LinkedIn Post announcing I am starting a new position
I’m all about the stories, so if you’re reading this because this type of thing is something you’re considering, please get in touch via the contact form or LinkedIn and tell me your tale!
I’m now going to publish this post with no review, second opinion or Yoast. So its back to mediocre blog posts I’m afraid folks, but I think for me, that’s exactly how I like it 😉.
Do you want to learn more about using Postman’s feature Flows, but are stuck for inspiration?
Want to see how it looks and feels in the latest version released just last week (late Jan 2021)?
Then have a gander at the livestream I did with the Postman team. In the video below, myself, Arlemi (Senior Dev Relations at Postman) and Vikram (Developer Advocate at Postman) :-
Create 3 basic Flows from scratch 🎓
Talk about how Flows has evolved and show off the latest improvements 😎
Let the cat out of the bag to say when flows might come out of Beta and into general release 🤫
Video of livestream of Postman Flows feature
And if you want to read more, take a look at my earlier posts on Flows here:-
I’m very *very* excited to finally be able to say it out loud. As you may have heard, I’ve now left my previous role as Staff Quality and Test Engineer at Smoothwall.
I’m genuinely thrilled to announce that for the next 2 months I’m going to be helping the team at Michigan based email testing specialists Mailinator as their QA Relations advocate. 🎉
I’ve known CEO Paul for awhile and when I mentioned I was taking a few months off he reached out to see if we could work together – and there was no way I was going to turn this opportunity down. ✨
I’ve since met the Mailinator team and can confirm they’re a lovey bunch, their product is legit amazing (I couldn’t back it in all good conscience if it wasn’t) and I’m itching to get stuck in!
Who are Mailinator?
I’m going to be writing a lot more about Mailinator over the coming weeks, but here are some of the systems features I’ll hopefully get to show off.
Key Features of Mailinator
Ever got stuck needing to automate a signup process that asks you to click a confirmation link sent via email? Or maybe your end to end tests fail because you’re asked for 2FA verification when logging in and you can’t automate that? Or how about testing the SMS messages your system sends? If so, I’m here to show you how Mailinator can help to fix those problems.
So many things to talk about, just hope I have time to do it before I plan on returning to the world of testing in March.
So yeah, if you can’t tell, I’m just a tad excited about this particular gig. Please reach out to me on Twitter or LI if you want to hear more – note I’m not a sales person, my job is to start conversations and share my knowledge. Let the adventure begin!
Beth The Tester's Tales 