Interviewing - Question at CAST2015

So I tuned into the CAST live stream again tonight, and I caught the end of Rob Bowyer's talk titled "Why Should I Hire You?", because I am bloody awesome! Sorry I digressed there. From what I heard it sounded like a talk full of experience and insight. Again as per last night, I decided to take advantage of the CAST twitter printer and ask Rob a few questions.

I asked this question having done some interviews recently for a testing role requiring automation skills. Well, I see the same things all the time. They tend to read something like this.
Created and maintained an automation architecture for UI testing using C# along with SpecFlow, NUnit and Selenium WebDriver.
Rewrote their UI automation architecture from Selenium RC to Selenium WebDriver using the PageObject pattern as well as the Data Builder pattern.
Lets look at the first quote. BINGO! That's right, buzz word bingo, and there is at least a line there. Now I am glad to hear about the tools you are using, glad to hear they are popular tools, glad you told me what kind of testing you believe you architecture is doing. But why? Why did you create this, for what purpose? Why did you choose those tools over others? How did you maintain it, why did it need maintaining? For some listing the tools is enough, but for me, it's not.

Now the second quote, this is better, less focus on tools and languages and more focus on patterns. Patterns are transferable, so I am happy to read this. But it's still missing. Why did you rewrite it? Why did you choose those patterns, what do they offer you?

So those quotes are from my LinkedIn profile. I realised during some prep for a conference talk, that I am massively underselling my self on my CV/LinkedIn, note that I still haven't got around to updating them. But clearly, so are many many others. Hence my question to Rob. As when interviewing recently, I was reading similar and find my self, going "and what". Huh? Really? So?

I am some what interested in the tools you are using, the programming language you used, but what I am really interested in, what really makes me go, "oooooooo", is when you talk about:

  1. Why you are using automation?
  2. How it fits into your testing strategy?
  3. How you decide what to automate?
  4. How you designed your architecture, and why that way?
  5. Why that language over another?
You see, as I mentioned above, things like models and patterns, they are transferable, they can be applied to most languages, most tools. If you can demonstrate an understanding of designing a good architecture, an understand or where to use automation and how you go about selecting tools, then that is value to me, that draws my attention. Given a few hours I can learn a new tool. Given a a week or so, I can get the basics of a new programming language. In both scenarios, I can google for help. But if you don't know what to google for, well then your a bit stuck.  

If you agree, have a look over your CV/LinkedIn profile, are you selling your real skills, or just listing some tools and patterns? 

Using BDD Tools To Write Automated Checks != BDD

Using Cucumber, SpecFlow or similar tools to write your automated checks, is not doing BDD. Cucumber, SpecFlow or similar tools are not testing tools ¹. I believe they were designed to facilitate the process of BDD. Sure they could be accompanied by other tools and code to form a testing tool, but as this post will elaborate on, its important to understand what doing that means.

I am seeing a surge, well it was a surge at first, it's considered the norm now, I gauge this by using blog posts I see, message forums and job adverts. I probably first noticed it in 2013. This surge is people using tools that have come out of the BDD community for their automated checks, with the most popular being Cucumber and SpecFlow.

Is there a problem? Maybe, as always it depends on the context. But what I do have a problem with is people claiming that by doing this, they are indeed doing BDD. Now I should make it clear from the off, I am no BDD expert, I have a shallow understanding of it, read a few books, a few blog posts, never worked anywhere claiming to do BDD, however I have worked in companies were collaboration is strongly encouraged, if not mandatory, and my understanding of BDD is that it's routed in increasing collaboration. Does writing your automated checks in SpecFlow/Cucumber increase collaboration? Minimally if at all.

My next gripe is people using Cucumber/SpecFlow in their automated checks, well, just because! "Why wouldn't we", I hear them cry, shortly followed by "it makes everything readable". It could make it more readable, however sadly the way most implement (generalisation here), but also based on what I read about and have experienced, they don't write them in a readable way. Ok I should make this more clear, the "steps" may be very readable, but what the scenario is checking, not so much. But thats the point of writing scenarios in gherkin, right?

There is a great example of that in, Fifty Quick Ideas To Improve Your Tests by Gojko, David and Tom in the chapter titled "Use Given-When-Then in a strict sequence". It reads like this.
Given the admin page is open
When the user types John into the 'employee name'
and the user types 30000 into the 'salary'
and the user clicks "Add'
Then the page reloads
And the user types Mike into the 'employee name'
And the user types 40000 into the 'salary'
And the user clicks 'Add'
When the user selects 'Payslips'
And the user selects employee number 1
Then the user clicks on 'View'
When the user selects 'Info'
Then the 'salary' shows 29000
Then the user clicks 'Edit'
and the user types 40000 into the 'salary'
When the user clicks on 'View'
and the 'salary' shows 31000
Have you seen a scenario like this before? What is this scenario actually checking? It's not immediately obvious to me, you? I am not going to dive into detail of what it could be, based on the above, as the author of the chapter has already done a great job of that, so I recommend reading it. But a possible scenario could read as
Given an employee has a salary 'X'
When the tax deduction is 'Y'
Then the employee gets a payslip
and the payslip shows 'Z'
Which works better for you? The latter I would hope. It's clear what the scenario is doing. Its clean. Also in most cases where BDD may be being practiced, or the acceptance criteria is being written in gherkin, it's directly related to the story, and could subsequently serve as an education resource for someone wanting an insight into some features/behaviours. The same couldn't be said for the first example.

So why are so many people automating like the first example, where we don't have these advantages?

I have started calling the first example TestCase 2.0 - By this I mean, if the steps are written like such, someone without any technical skills, can come along and make some new scenarios, just by plugging some steps together. They never have to look at the plumbing underneath, it remains hidden. It's like feature files have replaced spreadsheets... step driven automation instead of keyword driven automation. Takes little thought, and subsequently returns little value.

It's understandable why someone would use it this way if such a framework existed, "what! I can just search for steps and create 100s of new scenarios, awesome!" Sadly it's not. Not going to dive into that in this post, but we need automation that supports us, creating scenarios for the sake of it, especially ones where the scenario isn't clear, in the long run, isn't going to help anyone. Also, the more you add, the longer you wait for your feedback, you know the fast feedback that automation could provide.

I am not saying don't use tools such as Cucumber/SpecFlow in your automated checks, but think about why you would.
  1. What's the advantage?
  2. Where's the value coming from? 
  3. Who is your target audience for reading your scenarios?
But be clear to yourself and your team with what it is you are doing. In most cases you are using BDD tools as part of your automation framework, not doing BDD, so don't fool yourself or others into thinking you are.

So do use Cucumber/SpecFlow, but use it for the right reasons, use it to make it clear to you and other what the scenario is checking. Not because it's the "cool" tool at the moment.

If you still think what you are doing is BDD, then I have listed some references at the bottom.



Look up Dan North & Liz Keogh's work on BDD.

Testability Question at CAST2015

I was watching the live stream of CAST2015 earlier, in particular I was listening to Maria Kedemo talking about "Visualising Testability". Having done her's and Ben Kellys workshop at LetsTest, I was interested to hear Maria talk about this topic again, and also to see if anything from the workshop was in the talk.

Wanting to get involved more, I posted a question to the twitter printer, which on a side note, is an awesome idea.

If I could rewrite it, I would write. "Should testers with coding skills focus some of their time on increasing testability of the product/testing, instead of focusing on creating automated checks, which I believe is where the majority spend their time" sadly, that doesn't fit in 140 char.

I believe they should, as I believe automation is just a tool. The most common use being reducing testers time spent checking, by automating those checks. Which I also believe is where most focus their efforts. However some testers skills are on par with some developers now, especially those occupying the role of SDET and such. So surely we could use those skills to increase some aspects of testability. For example, referencing James Bach model, someone with those skills could spend time improving intrinsic testability, altering the product itself. This could be adding in some logging they require, it could be writing some hooks to make accessing the system easier and much more.

But for me, I want to see more testers focus on what James titled "Project-Related Testability". I encourage people with coding skills, testers or developers, to create tools that really support the testing efforts. For example, they could write tools for reading logs files, creating data, state manipulation, data manipulation and much more.

Of course with any automation pursuit, it should be clear what is trying to be achieved and be aware of falling into the automation trap. If something it taking to long, will the value be returned by continuing or should you just accept defeat.

Anyhow, I encourage all to watch Marie's talk once its published on the AST YouTube page and think about what tools could you create, or someone in your team create, to increase an aspect of testability.