Whiteboard Testing Has Arrived

So, a while back whilst researching a testing topic, I was searching YouTube for relevant videos. I found myself in this circle of thinking I had found something, then realising it was awful. Finding something different, but then realising it was an hour long. I was appalled at the quality of testing related videos on YouTube. Only good content I was finding was conference talks, but again, all lengthy.

So this nagged me for a while, I started thinking, I am sure a lot of people must turn to YouTube like I did, so therefore they are being faced with the same poor videos I was finding. This isn't good for our craft. Especially if people are new or less aware about testing then I am, they make take some of these videos as gospel.

So I decided to do something about it. I turned to my trusty whiteboard in the office and started scribbling, mapping out some ideas. Then it clicked, what if I recorded some video's in front of this board, using the board as "living" slides, so to speak. I could draw up models and critic them, I can write out key bullets point and add to them during the talk, I could pretty much do anything.

So there it was, I had created Whiteboard Testing. You can checkout the YouTube channel and I also created a twitter account for it @WhiteboardTest. There are currently two videos uploaded, one is about my plans for the channel, and the second is about a model I drew in relation to Regression Testing, the FART model. You can find them by clicking on the channel page.

I explain this in the introduction to the Whiteboard Testing video, but I want to repeat it. This channel is not for me, it's not about me. The goal for the channel, is to fill it will short and relevant information on Testing. I am talking videos that are 5-10 minutes long (yes I know, the regression one was 12 minutes, I will get better :D). But, I can not do this alone, I need the help of the wider community.

So if you have the following, which I'm sure you all do, consider creating a video, send it to me, and will can look at getting it on the Whiteboard Testing channel.

  • A whiteboard, flip chart or that sticky stuff you put on the wall.
  • A video camera or a smartphone.
  • A tripod, or a friend to hold the camera. 
  • Something to talk about.
These videos are not high production, look at the intro video to work that out. I'm not looking for fantastic visual effects, I'm looking for awesome content, that is going to help drive our craft forward. 

So please do help me share awareness about Whiteboard Testing, and I looked forward to seeing some of your videos on there in the future.

You can watch the intro video here


Tackling Android Deployment Using ADB and Appium

Problem

It's taking me a bout 10 minutes to update all my Android devices each build. Sometimes, I have 3-4 a day, so that's 30 minutes I am not testing. Also, installing the app is really boring. Then I have to log into to the app on all the devices.

This was the problem I set my self, I solved the problem multiple ways in the end, a journey I felt worthy of a write up.

State of play

We are currently putting new .apk files in dropbox/google drive. Then I either install that apk via Android Debugger Bridge (ADB), or I use dropbox/google drive on the device and install it straight form there. Then I proceed to log into the app on all the devices. As mentioned above, it takes me about 10-15 minutes to do all the devices, it's rather boring.

So I thought, there must be an easier way to do this. So I explored, and created multiple ways, so here goes.

Bash Command

My thinking went a little like this, if I can use ADB to install the apk via the Terminal, surely I could connect multiple devices, create a loop, and install many at once? Seems sensible. Turns out you can. So I brought myself a 4 port USB hub and turned to my best friend, Google.

After a few minutes googling, I stumbled across this command
adb devices | tail -n +2 | cut -sf 1 | xargs -I {} adb -s {} install <Path to apk>

In short, and I am no bash expert, this command takes the response from "adb devices", trims the response to just get the device names, then runs "adb -s install <Path to apk> on all the results.

Fantastic, I could now install the app on as many devices as I could connect to the laptop. 

But what about opening the app and logging me in? Well I knew I could start the app using ADB, I had seen such a command appear in Appium logs when I was using that in the past, so again, a quick google and I had the appropriate command 

adb shell am start -n com.package.name/com.package.name.ActivityName

Of course replace the package name and activity with ones for your app. If you don't know them, ask your Android developer, they will know for sure. 

So we are on a roll now, 15 minutes in, and I can now install and start the app on as many devices as I can connect. Excellent. 

So what about logging in? Well unfortunately, that isn't possible via ADB, unless I was able to write some code to interact with the UI, compile it, and send it to UIAutomator, hmmmm, sounds very much like what Appium does.

Appium

So I've used Appium a lot in the past, not so much these days as I don't believe UI checks are worth the investment on mobile, the tools, including Appium, just aren't mature enough yet. That isn't the tools fault, the platforms themselves are only around 5 years old, think how long it took for browser to be stable enough for automation? Some would argue they still aren't.

But I knew in theory what I wanted to do, was perfectly feasible in Appium. So I set about making it happen. 

Here is an insight into my approach for this. I knew what I wanted to do, so I wrote that out as code comments, like so:
  • Find out how many Android devices connected.
  • Get there IDs
  • Start Appium
  • Run a script that installs the app, starts it on all devices
  • Get the Appium to log me in

So I knew that ADB would tell me how many devices were connect, so I needed a way of executing a bash command from Java, again, hello Google. Discovered multiple ways of doing it. I could then read the output of this command. Then as per the ADB command from earlier I needed to trim the responses so that I was just left with the devices IDs. 10 minutes in, first two steps done.

So, as Appium is based on the same API design as WebDriver I knew how to create a AndroidDriver and pass in the device ID and path to my .apk, I also knew that Appium would attempt to launch the app for me.

Then I also knew, that I needed to tell Appium which activity to wait for, again if you are not familiar with activities in Android, I suggest reading about them. Then I just needed to automate the login screens and job done.

So I had been going, about 30 minutes now on my Appium implementation, all was going really well, then things took a turn. It turns out each Appium server can only have one active session, meaning that it can only interact with one Android device, I am not sure why this is, potentially something to do with ADB connections, I am not sure. This was based on me actually trying it, it failing, googling a bit, and seeing others complain of the same problem.

Appium Workaround

So I needed to find a workaround, some of the responses on Appium's Discuss site, suggested a quick solution, this was to start multiple Appium servers, essentially one per device. Well as above, I knew I could send commands via Java, so surely I could just do the same for starting Appium? Turns out, you can. However, they appeared not to have access to all the alias set on my machine, specifically, ANDROID_HOME which was causing issues for Appium. Sigh.

We are about 45 minutes in at this point, and I came across this excellent code example by Prithivi on this discuss page, this appeared to do something different, I am not 100% sure what the difference is, one observation, is that I can now see the Appium console within IntelliJ so I am guessing it runs the command line arguments within the context of my application, not sure, and right now, it's not that important to me. 

But it worked! I could now systematically start Appium servers. I was now able to start an Appium server per device. Now, in order to do this, you need to set a different port for each server. So looking back over what I had so far, I now had two parameters per device, device name, and a port. So I decided to create a device object, incorporating those two values, so I can pass these objects around.

Update: Copy and paste error from an old project of mine, I since updated the solution to Appium Java-Client 3.2 which comes with ApiumServiceBuilder support, so the above solution is not superseded and far easier using this, I will update GitHub repo. 

Improvements

Right now this tool, is a huge leap for me, and already a week in, it's saving me a substantial amount of time. However I can see some improvements in the pipeline already.
  • Can I thread/parallelize the Appium call, to make it do all the devices in one, currently is sequential.
  • The apk file is currently hardcoded, seeing as we have a naming convention for them, could I pull the latest .apk from Google Drive to make it dynamic. 

Conclusion

So all in all, this was about two hours of work (additional time comes form creating the login code), but I do now have a tool where I can deploy and login on multiple devices. 

I uploaded the basic code I "stole" and wrote to a GitHub repo. I don't proclaim to be an excellent programmer, so if you like this and can improve it, please do, I would love to see it. 

Insights

I posted this blog for a few reasons. The first being that, I think this is an awesome tool. A fantastic example of what I (and others) try to tell people about automation, it's not about trying to automate checks all the time, identify a problem and attempt to solve it.

Secondly, it's OK to steal other people's code. Just don't try to pass it of as yours on a public repo/blog, thank the other people. But off the back of that, take advantage of the knowledge that is out there. Google all the things. Look to the issue and discussion pages for the tool you are using, especially with open source tools. 

Finally, I wanted to reflect and share how I approach such problems, hopefully some of that came across.

Thanks for reading, be very keen to hear from others who use this, or have other solutions. 

Yousaf Nabi - Ermahgerd TestBash New York City Y'all

So about a month ago, a ran a competition to win one of five tickets to TestBashNY. Part of the deal of entering the competition was that, if you won, you would agree to write a short post/article about your day at TestBashNY.

Here is Yousaf Nabi talking about his TestBashNY experience. You can follow Yousaf's future journey on twitter here

5am Wednesday morning and my alarm rouses me from my blissful slumber. I fumble around for the snooze button and nuzzle my head back into my pillow. 10 minutes later and my eyes open again to an Englishman’s morning vice, a steaming cup of tea graces my bedside cabinet. (Thank you Helga!)

For those who know me, will know that 5am has been my bedtime more often than its been my wake up call, so to arise at this ungodly hour, it must be something awesome. Nope, its not a car show (for once!), its TestBash. For those who aren’t aware, TestBash is a unsurprisingly awesome testing conference usually held in Brighton, but for the first time it crossed over the atlantic to be held in New York City.

An uber and train ride later, I find myself on board AA2201 bound for NYC. Now we all know about the time differences when going across the pond, and the key advice from all those I spoke to was, sleep during the flight so your refreshed when you arrive. Sleep, baaaah humbug. I’ve got a test suite for build for a new node based API that is currently under development. A true tester never switches off, after all its a mindset not a job! (More on my fumblings through the world of greenfield projects in an upcoming blog post).

So with my first day free, I spent it wandering the streets of New York in 21 degree heat (there goes the winter clothing!) and made some observations of which I feel it pertinent to share, if only to get a little insight into my brain.
  1. People don't queue, or get annoyed when people push to the front. How??? 
  2. 
No one cares about the police. People just cross the road in front of cop cars with sirens on, totally nonchalant about it too.
 
  3. Pedestrians think they have two wheels since they all walk in the cycle lane. Cyclists don't mind - why so polite? I'd be raging.
 
  4. There don't seem to understand steak sauce - what the feck is A1 sauce?
 
  5. There are no kettles in hotel rooms. How is an Englishman meant to have his tea?!
 
  6. I am jealous of every house in the suburbs as they all have mahoosive garages.
 
  7. Taco Bell didn't live up to my expectations. (I forgot that I ate Taco Bell at the Arndale in Manc and felt the exact same way) 
  8. Generally everywhere is pretty grotty and dirty. NY certainly scrubs up well in the cinema.
    Day 2 – Workshop Day.

    After a brief walk to Baruch College, I transcended the elevator to the 14th floor to be greeted with some happy faces wearing Ministry of Testing. WOOHOO, I’m in the right place, and OMFG they have tea. Morning sorted, now for some “Test Automation & Continuous Integration at scale with Dr. Jess ingrassellino & Noah Sussman.

    The day kicked off with a backstory of how Jess & Noah came to be where they are, and a disclaimer to state that they are not professional speakers, but the well attended room was very receptive. Everything is better when it’s a bit rough around the edges :)

    After a brief dialogue of the pain-points of automation testing, it became clear (as we were running out of whiteboard space), that there are a lot of issues out there, which affects software testers of all walks from all industries. The pain is universal, which can only mean that where people have succeeded building successful automation, that we can learn from their achievements to help each of us in our respective jobs.

    The future talk by Tanya Kravstov from ROKITT at the main conference would lay testament to that. I subsequently fed-back to Jess that some of Tanya’s high level points would have been a good structuring and discussion points for the test automation portion of the talks.

    Noah broke up the discussion with his sand-pile game, which was analogous with system building & change management. It was very interesting, throwing up lots of varied conversations about the various ways of implementation and the risks of each, much like the real world. One of the take home lessons from that was, if you plan ahead and plan well, and accept that there is allowable level of loss, you can provide far greater coverage (near full coverage) and have a relatively non-complex system. I do have to admit that I found a defect in the game instruction slides and could not stop myself telling someone. I’ve never been very good at keeping secrets either =/

    Noah then went on to discuss his devOps experience, and although some of us attending have devops departments who already implement graphite/grafana, I had never considered how it could be used as a testing aid in CI for providing monitoring and quick feedback loops. Coupled with a raspberry pi and a monitor, I now have the initial knowledge to allow me to output metrics directly from the code within my test suites, which even for the purposes of debugging will be so useful in my day-to-day job.

    I’ve jotted down some of my notes below, mainly because it’ll help me when I come back to implement it in the new API suite I’m testing. If anyone wants to explain any of the points further, please ask me, or indeed Noah S. as I’m sure he would be very happy to talk about his experiences.

    Some command line techniques for log trawling
    • Download your log
      • http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html
    • Unzip your file
      • gunzip NASA_access_log_Jul95.gz
    • Word count on the log
      • cat NASA_access_log_Jul95 | wc –l
      • wc -l NASA_access_log_Jul95
    • Echo to the command line, all the log lines with 200 in them.
      • grep ' 200 ' NASA_access_log_Jul95
    • Echo to the command line, a count of the log lines with 200 in them.
      • grep ' 200 ' NASA_access_log_Jul95| wc –l
      • grep -c ' 200 ' NASA_access_log_Jul95  
      Example code to publish data to graphite
      • Example pseudo-code that will execute your log trawl on the command line
        • echo $(wc /-l NASA_access_log_Jul95)
        • echo $(cat NASA_access_log_Jul95 | wc -l)
      • Extend the above by adding a timestamp
        • echo $(cat NASA_access_log_Jul95 | wc -l $(date +%s)
      • Extend the above by sending it to graphite as my.nasa.metric
        • echo my.nasa.metric $(cat NASA_access_log_Jul95 | wc -l $(date +%s) | nc graphite 
      Some take-home points from the Noah’s talks:-
      • Piping to | nc graphite we can publish metrics to graphite for consumption.
      • You can do any amount of pre-processing to the data before publishing it to graphite.
      • You can put this all over your test code, you can limit the extra traffic it will generate later.
      • You can put statsd in between your code & graphite in order to choose only certain amount of requests to sent to graphite and additionally aggregate stats from several sources (ideal for load-balanced systems).
        • You might not need every message, or want every message (think DDOS).
      • Use UDP for messages as it is not stateful unlike TCP.
      • You don’t want your code waiting because it’s waiting on an ACK.
      • You can use XARGS to build up a list of commands which can be run concurrently
        • The following example that will execute the following command as 24 concurrent processes (but it could be your PHPUnit tests etc)
        • cat NASA_access_log_Jul95 | xargs -P 24
      Regex – If you aren’t using it somewhere in testing, then you probably should be. Once you get the hang of it, the possibilities are endless. From testing fields in an API response (dates for example) to grepping your logs to pull out all sorts of information to suit your particular needs. It’s a skill not many people have really mastered yet. There are some great tools out there that I use to aiding me in my regex.
      I love them both and use them a lot. Glad to see Noah was also a fan of regex.

      Overall, the scope was huge and there was so much to cover, which was probably unfeasible for a single day workshop, but it was thoroughly enjoyable, informative and useful. Massive thanks to Jess & Noah for taking the time to run the workshop. You guys did great :)

      Day 3. TestBashNY

      First off, how freaking cool was it to see TestBashNY text up on the on the gramercy theatre billboard. I bet Anna & Rosie were properly made up.

      Secondly, never seen so many testers in one place before!

      It was amazing to see such a diverse range of people from all over the world and that they had all taken time to either attend, participate or present.

      I could talk for days about the thought-provoking, engaging and funny talks that everyone put on, but I hear they will be available on the Ministry of Testing Dojo as eye-candy for your pleasure. You lucky people, as they were all brilliant, I would definitely recommend checking them out on your lunch break, while you work, while you poop, wherever!

      The whole event showed me that the testing community is in great shape, with many proponents from which I have been inspired to do the same. It came at the right time in my career, where after 10 years of breaking stuff daily, I was feeling a little lost. Knowing that this bubbling cauldron of knowledge is there at my fingertips is a great help and I know that I can help add value from all the pain I have endured over my time in testing.

      Sometimes it feel’s like you’re the glue that holds it all together, if only to be the gatekeeper. I remember the days I used to be reluctant to take holidays for fear of renegade developers with specification allergies running amok. It makes me laugh whilst reminiscing but it was hectic at the time and my mouth was rather colourful to put it nicely.

      So with that new found enthusiasm, I am going to get more involved with my local testing community bringing testing into the focus of the wider technological talks we hold at Sky Betting & Gaming. I’d love to open an invite for any speakers in the testing community to present, and anyone to attend.

      It’s been one of the visions of our company to be one of the best digital companies in the UK and we believe that becoming a technological hub is part of that. Testing hasn’t really had much focus in our past meetings so I am definitely seeking to redress the balance but I can’t do it without your help. Who knows, we may be able to hold a TestBash up int’ Yorkshire. Pie’s n Pea’s at the ready.

      Herman Ching - Caveman first painting

      So about a month ago, a ran a competition to win one of five tickets to TestBashNY. Part of the deal of entering the competition was that, if you won, you would agree to write a short post/article about your day at TestBashNY.

      Here is Herman Ching talking about his TestBashNY experience.

      Test bash, was great in it’s mature offering and highly accepting of the seemingly young community of testers. After the conference was over and a few drinks, I felt satisfied that the conference was worthy of my time. I will be attending my first few meet ups in New York city soon.

      It was my first time at testing conference, my first time writing anything published on a public blog and my second time writing this. I walked in waiting and sitting in a chair in a dark theatre. With time master Mark on the mike, I was ensure a very prompt schedule.

      From collaboration to profanity, it was a wide range of offerings. I could make a few recommendations but I am going to hold onto those. I learn a few things I will be taking back to my work but I am not going to share those either. I want to eliminate bias which I did have a lot of the first time I wrote about my experience a test bash. Let this conversation set the tone and hopefully that alone will entice you to review the content that was offered if not the mysterious I seem to have just created should grow some curiosity.

      The talks set a tone I immediately could not relate to until the plethora of 99 second talks. Someone asked me half way through during the break, what did I think of the talk. I did not have an answer. Normally my immediate go to response was “it was good, I am learning a lot”. I kept that aside and instead said “I wasn’t sure yet, still a lot of content to digest”. One could easily recommend just view the videos/presentations online once it was over. I had to take something away from this conference that static could not offer. Time to start talking.

      Beyond personal stories from audience or advise from the speakers, I could say my biggest inspiration came from the concern, the lack of exposure the general community had around testing. Being a father of two, I could relate to Anna who wanted to created more exposure. Additionally from Helena seemingly water colored presentation and Selena growing hooks into the community, I want to give back. I am inspired to make a children book on testing and I hope to make it available to the testing community. This whole conference entice me to do just that. I could only imagine what’s it doing it for everyone else.

      Melissa Eaden - TestBashNYC == Mind Nova!!

      So about a month ago, a ran a competition to win one of five tickets to TestBashNY. Part of the deal of entering the competition was that, if you won, you would agree to write a short post/article about your day at TestBashNY.

      The first of those is in! Here is Melissa Eaden talking about her TestBashNY experience. She also decided to join twitter whilst at TestBashNY, so you can now follow her testing journey here.

      I could have used mind blown, or mind explosion, but really, it was a day of brain shaking presentations, of foundational changes to my testing habits and philosophy, on epic, but very personal proportions. It was like an internal nova happened in my head. I could not look away; I could not look back. I cannot continue to test as I have been testing.

      Even the paragraph above doesn’t really describe what exactly happened to me on November 6th. I attended another conference for testing before TBNYC. I spent three days at that conference and walked away with one page of notes and freebies. Lots of freebies from people that were mostly trying to sell me something. I vowed to never go to another conference like that again.

      My essay to Richard Bradshaw was a last ditch effort to get to this conference. When people asked about it I described the essay, in short, as a story about wanting to have Ministry of Testing’s love child and how going to TestBashNYC was absolutely like giving me an all access pass to my biggest professional/testing crush.

      I am absolutely pregnant with ideas! My head was so full from the conference, from the conversations with people, from my own ideas on how to implement things I learned, I’m not even sure where to start with them.

      All the presenters were wonderful! The conference format allowed for conversations with presenters that normally don’t happen at bigger conferences either because of time constraints or just from having too many people lined up to ask questions.

      Everyone was there to learn something. I walked away from that conference with ten pages of notes, and no fewer than five business cards. There were so many testers and even developers there, heading the right direction and doing the right things and encouraging loud mouths like me to speak up more and do more in the community. I was never discouraged from my opinion or view, but more often than not, I was gently led in another direction, or given a viewpoint I hadn’t considered before. Or even more to my surprised, agreed with, often. I didn’t have to fight an uphill battle. It was the greatest example of culture fit happening right before my very eyes.

      The biggest takeaways for me were to follow my fears or my dreams and co-create everything! Collaboration was a big theme throughout the day and into the discussions later that night. I’ve been asked to speak, write and even help with a podcast. I don’t think I would have ever been offered those opportunities at another conference.

      For as long as I am testing, I will be forever a fan of Ministry of Testing and their TestBash conferences.