Tail Log File And Play A Sound When Keyword Hit

So on the testers.io Slack team today, Maik messaged:-
Does anyone know a way/tool to watch a log file and play an audio sound, if a certain keyword is found (e.g. ERROR)
there is "log watch" for windows from James Bach, but I am looking for a Macintosh
Having only had a mac for about 8 months now, I have been using the terminal more and more. I immediately thought upon reading Maik message that this must be possible. Now I knew how to tail the log file but I wasn't sure about playing the sound. Turns out on linux you can use aplay, however doesn't seem to exist on the mac, but a few googles more and I discover that the mac has afplay. I simply googled to find a beep .wav file for use with this.

So I gave it a go, and success was had!

Now the funny bit, well wasn't funny at the time, but it is now. As I was testing this, I created log.txt and was editing it in Sublime and the sound would play when I added "ERROR" to a new line in the file. Success? Not quite, it would also beep for every "ERROR" in the file. Unable to work out why this was, I told Maik I was done, but informed him of this "feature". Decided it was a feature and not a bug, he could it use to see how the day had gone. If the final error beep of the day lasted for a few minutes, well then that would have been a day full of errors!

But I was left feeling unsatisfied with my work. So I broke the script down and tested it in individual sections, the logic was sound, it was doing what I expected it to, but not working as per my goal.

Turns out, Sublime doesn't append text, instead it overwrites the existing file! So my script was behaving correctly when it played the beep for all the errors thus far, because as far as tail was concerned all the lines were new! So I decided to see how I could append a new line to an existing file without overwriting the whole file. Turns out its actually quite easy.

echo "sdasdsa" >> log.txt 
The above command would do this for you. So I tested using this, and low and behold, only beeps when a new line is entered containing "ERROR" and it only plays once.

So there you have it, a fun 30 minutes was had by all. I learnt some more about shell and using the terminal and Maik got a script, that he later confirmed works for him.

I never asked Maik what his particular requirement was for this tool, but its something I have done in the past before, and as Maik mentioned in this original post, James Bach built "log watch" for windows.

Log files are used to capture unexpected errors and to provide information on the product, information that is gold to a tester. So why not monitor it in real time, and having a sound play on certain matches is a nice way to get your attention. Can be very useful when in a testing session, as the beep may alert you to something you hadn't seen, but caused an error. For instance the UI may not report any problem at all, but being alerted to an error, may mean you actions had actually caused this, but you would have never known if not monitoring the logs. Reading them after the event would alert you to the fact there were errors, but being after the event would make it harder to reproduce, to work out what caused the error. So using such a tool or script to alert you can lead to some interesting discoveries.

This is exactly the kind of thing I want to hear more about with regards to how automation is used within testing. For me this is a fantastic use of automation, the type of automation I talk about, automation that really supports the tester. We need more of these things. It's nothing new, but perhaps gets less focus in some teams due to pressure to produce automated checks. But this whole thing took me 30 minutes, I can only imagine, but in my experience of using "log watch" this will repay itself many times over for Maik and others who decide to use it.

Happy log monitoring. There was talk of playing the Benny Hill theme tune upon a match, this is of course completely optional :D but highly encouraged.

p.s. Here is a follow up post from Maik. http://hanseatictester.info/?p=566


An Introduction To The Data Builder Pattern

The data builder pattern was first introduced to me by Alan Parkinson, he did a talk about it at the Selenium Conference in Boston back in 2013. You can watch it here.

I want to share with you a current implementation of the pattern I am using, which was created by myself and Lim Sim. It not 100% aligned to the following links, but its working well enough for us.

So what is the Data Builder Pattern? Well we start with "Builder Pattern", officially its "an object creation software design pattern". So in the context of data, the pattern is used to create data in the system under test. In my words, its a maintainable pattern for creating and managing the automated creation of data in the system under test.

It's a fancy name for something that is surprisingly simply.

In it's simple form, the flow is as follows: Model > Build > Manipulation > Creation. We have a model of a domain object, we create an instance of it with some values, we manipulate some values (not always) and then we create this object in our system.

Model
This is a model of the domain object you are wanting to create. So we have to look at our domain and break this down into models. The application database can be a good place to work these out, in some cases, you could simply copy the schema. The code base is also a great place to look, and again, in some cases the models may already exist, allowing you to use them directly in your code or make a copy of them. If you don't have access to any of those things, just take the time to work it out. The application UI could also be a good place to work these out, but be aware that sometimes important values are never displayed in the UI.

So lets look at an example, we will use this throughout this post. I am going to go with a user of a system, its a made up system, but you will get the point. Should point out for this example, I haven't made the code fluent, you can read more about that here, it doesn't really alter the pattern, but as I said, this is how we have implemented it.

As you can see, keeping it very simple. We simple have a model of a domain object with getters and setters so that we can assign some values once we create an instance of it.

Build
I have called this stage build, this is the stage where we create an instance of our model. So in other words, we create an instance of our class (model), we create an object. We build the model.

We have called these builders, it's where we create an instance with some values. Those values can be generic or you could use a random string/number generator to create values for you. It all depends on your context. Or they could be scenario/persona based.

The builder is a class, that consumes a model. In turn it knows the context of the model and importantly what values to assign to its properties. The builder will then return an instance of the model with usable values assigned to its properties.

You could have a simple generic builder like so:

So you can see, all we are doing is assigning values. We have decided that in our context, those values fit what we would consider a generic user.
But of course we could now expand the builders available to us, to fit other scenarios that may exist within our context. For example an OAP user.

Notice that I added it to the same class, you should do the same if going to try this approach. This class is the builder for a specific model, so this is where you would continue to add your builders. So another common approach to data, is to just randomise it, as in some scenarios the data is irrelevant to the test/check you want to do. So you create yourself a builder that creates random values. You can do that in line, or if you are repeating them a lot, create yourself some utilities to create strings of all lengths and so forth.

So create a builder for all specific personas or scenarios you require. Of course you should consider if your builder needs to adhere to the schema of your database, for example, the firstName field in the database has a character limit of 50, so if the builder produced a 60 characters firstName, that wouldn't really work. So be aware of data limitation for your system.

Manipulation
So at this stage with have our object and it has values assigned to all its properties, it good enough to get created if we so desired. But sometimes we do want specific values, perhaps our check is going to look for a specific surname or search for users of a specific age. We need to be able to create data that is unique, but we don't want to to have to assigned values to all the properties, only the ones we are interested in for our check/test.
So what we do is make changes post the builder, so we let the builder assign all values, then we change the ones we need to before we create. So it would look something like this, where I need a user with the surname "Richards" and they need to be 35.

So as you can see, I only change the values I need and the rest of the values remain as they were from the builder.

Creation
So as with the build stage we needed a builder, with the creation stage we need a creator. The job of the creator is to actually create our object in the system under test. How your creator creates the data is solely dependent on your context and the testability of the system under test.

Some ways that you could do it. You could go straight to the database and insert your data there. You could use APIs if they are available. You could even use the UI if no other alternatives are available or if the context leans in that direction. There is no right or best way, it's whatever works, but by following the pattern, you allow yourself to change it in the future if a faster or easier method becomes available.

So what does the builder do, the builder basically takes your object, reads all the properties of the object and uses flows that you create to insert that data into the system under test, resulting in the data being available for your test/check.

I am not going to show an example of a creator as they are hugely context dependent and am sure you understand the job of the creator.

Summary
So once you have your models, builders and creators, you will end up with something like this, that would result in data being added to your system under test.

As you can see, this is really easy to read, thats because I have used names specific to this domain, sure in this example its just "User" but am sure you can imagine how your domain may look. Then we have used nouns for our objects so there purpose is clear to us, and verbs for their methods, create and build.

Anyhow, I hope this was a nice introduction to this pattern if you were not familiar with it. I will follow this post with a few more on this topic. We look at more advanced implementations, as well as how following this pattern allows you to leverage the code for multiple uses throughout your testing approach.





How Often Do You Really Fix A "Failing" Automated Check

Do we really fix a failing automated check? Or do we simply defunct one and create a new one.

I saw a tweet the other day from an old colleague of mine.
It got me thinking, how often do we really fix a failing automated check? By fix in this instance, my thoughts started with getting it passing again, getting it green. Even though I prefer not to talk about passing and failing, for the context of this post, passing means satisfying the algorithm and failing means didn't satisfy the algorithm.

Lots of discussion followed on Twitter after I tweeted, "You very rarely 'fix' a 'failing' automated check", https://twitter.com/FriendlyTester/status/592972631705559040, but going to try and summarise the thoughts.

So lets run through an example, I have a sign up form that has 10 fields on, I also have many checks that are all passing on this form. A new field has been introduced to the form, a mandatory field, this has caused 5 of the checks to now fail. In order to get those checks passing again, then need to be instructed to populate the mandatory field. Does this class as fixing the check?

I don't believe so, for me this is a new check, it just so happens to reuse 95% of code from a now defunct check. So we haven't fixed it, we have created a new one. Semantic argument some of you may be thinking, but I don't think so. For me, its a very important distinction.

How often have you been in this situation, the build goes red, someone investigates, 5 checks are failing, then you hear "fix those checks!". If you haven't, replace check with test, and try again. It's certainly something I have experienced multiple times.

Someone then runs the checks locally and immediately sees that the new field isn't being populated, "oh look it's not populating the mandatory field, I will make it do that", they do it, run them, they all pass! Job done......

Here lays the problem, how do they know that mandatory field is even meant to be there. Well likelihood is they did know, which then leads to the question, why weren't they "fixed" as part of the initial development work? One problem could be the separation of coding the feature and creating automated checks, happens in a lot of places, especially where 'testing' is a separate activity. It could be they run them after and then fix where required. But I feel it's because teams don't stop to think about the impact changes will have on the checks upfront. Once the work is done, the checks are ran like a safety net, failures investigated and 'fixed'.

So what's my gripe here? Well I feel more people need to give their use of automated checks more focus, you could write the cleanest automation possible, but if you don't know what they are checking, if anything at all, what use is that to you? It's like the building the wrong product right. We should be thinking at the time of developing new features, what new checks are going to be needed here, and importantly what checks are not. Are existing checks now checking enough, what else should they be looking at.

Checking can be an important part of an approach to testing, evidently very important at companies where there have created hundreds or thousands of automated checks. If you are going to rely on automated checks so much, then you need to apply a lot of critical thinking to what checks you create and continue to maintain. As repeated many times, your automated checks will only check what they are instructed to, they are completely blind to everything else.

Designing effective checks, by effective I mean adds value to your testing, supports you in tackling your testing problem can be a difficult process and requires a lot of skill, some are more obvious. It isn't something someone who doesn't understand testing should be doing. Now, turning those into automated checks, sure, I could see that being done by someone who doesn't understand testing, it would be far form ideal though in my opinion.

The reason I say it's far from ideal relates to this tweet from the start of the year.
Now of course this depends on your framework of choice and the architecture you are building on, but the creation of automated checks can be one of much exploration. It can also be the case of adding some keywords to a spreadsheet, but even in that scenario much exploration would have already been done.

The application needs to be explored at a level you may have not yet done. In the example of the web, the html needs to be explored for testability, is this a site I can easy automate? At the API level we may discover json elements that we have no idea what they do, or where they come from, we need to work these things out, we need to test.  Also as we are automating a check we may become aware of other attributes/values that we should be checking, and adjust the check accordingly, again though, this requires thought, requires skill. There is also the process of testing the automated check, something I have previously written about here.

Feel myself going slightly of topic, so lets try to wrap this up. Testing encompasses checking (James Bach & Michael Bolton, http://www.satisfice.com/blog/archives/856), your automated checks should be supporting your testing, helping you tackle your testing problem. Regularly review your automated checks, don't be afraid to delete them, always be evaluating there effectiveness. If you find yourself 'fixing' lots of automated checks, take the time to stop and think about what you are really doing.
  1. How could the situation have been avoided?
  2. Could it have been done earlier?
  3. What is this check even checking?
  4. I have "fixed" this many time already, is this an effective check?
  5. What information is this checking giving me?
Don't chase green, chase automated checks that support your testing problem. Don't blindly "fix" automated checks. Also for another post, something that we discussed in my automation workshop at LetsTest and Nordic Testing Days recently, do you checks clearly indicate their intention? Sure we can read the code and see what it does, see what it is checking, but what about its original intention, where is that? Be writing about this soon, your "fix" may actually break the algorithm, and therefore mis direct testing effort.

Remember the code isn't the check, the check is the algorithm. The code is the implementation of it. They may not always align, especially over time and with lots of fixes. Focus on both.

P.s thanks to Maaret and Toby for their posts, here and here respectively. I intended to think about this more back at the original tweet time, their blogging gave me the nudge needed.

p.p.s I should add, that I believe its OK to say you are fixing check if you are changing the implementation of the algorithm, as long as those changes don't alter the original algorithm. Such as changing the data or updating some locators. Or even the url's. Things along those lines.