Tag Archives: SIGiST

SIGiST Israel evening conference: Free and OpenSource Testing Tools

Last week, I attended the latest SIGiST Israel evening conference. SIGiST Israel is the Special Interest Group in Software Testing, Israel, and they hold evening events about 3 times a year, a 4-day testing conference once a year, and other test related activities every now and again.

The subject of last week’s conference was “Free and OpenSource Testing Tools“, and there were 3 presentations, together with great food, and a lot of networking.

The evening was kicked off, as usual, by the very competent Alon Linetzki, the founder of SIGiST Israel, who introduced the 3 speakers, and updated the audience with details of the 2009 summer conference, which will be held from the 29th June to 2nd July, focusing on “Adding Business Value and Increasing ROI“.

There will be 2 days of workshops followed by 2 days of track sessions. Some of the workshops and talks will include TPI, Agile Testing and Performance testing, and among the speakers will be international names, such as Andy Redwood and Mieke Gevers.

Dudu Bassa, of Sela, one of the SIGiST’s biggest supporters, made a few announcements relating to test engineers and test managers (if you are one, looking for a job, or your organization needs one, you can contact him).

Dudu also announced that Yaron Tsubery yaron tsubery, the ITCB President, is running for Presidency of the ISTQB, and has a good chance to be elected. If he is elected, it will be a great success for Yaron himself (of course), but also a vote of confidence for the whole testing profession in Israel.

Yaron – we wish you the best of luck !!

OK, so back to last week. 

SNAP, by SAP

The first lecture was given by Asaf Saar, QA Manager at SAP, and initiator of the SAP Netweaver Automation Platform (aka SNAP). Asaf described the complexities that his team has to deal with:

  • multi-site development and integration, including tests developed in many countries
  • many technologies and frameworks, including frameworks for black box and white box testing, developed in house – each office having developed its own … 
  • different testing techniques – manual, automation, unit testing, and API testing.
  • more than 10,000 UI automation scripts, and more than 100,000 Java unit tests (we should all be so lucky!) 
  • too many solutions, that can’t be synchronized
  • lack of usability
  • need to manage the Life Cycle  – run it all, get reports, analyse the reports, and currently each type of report sits somewhere else: XML, database, access, automation report, etc.
  • all tests needed to be in the SAP test repository, 20 years old, with millions of tests from the whole world, and no API …

In short – unmanageable!

And from the unmanageable grew the idea of SNAP, which was developed by just 2 test engineers and a student over a 10-month period, in parallel to their regular work!

How did they do it?

SNAP was developed based on Visual WebGui, an open source RIA (rich internet application) development & deployment platform, atop standard .NET. Visual WebGui, from a small Israeli startup in Kfar Saba –  Gizmox – enables development & deployment of applications on the server which are then virtualized on a standard browser with no specific installation.

The development was done in C#, compiled, and out came an Ajax web application!

SNAP itself enables integration of any testing framework via API/Web services (test drivers), and is used by developers, test engineers, and integration engineers. The actual tests are run on temporarily idle PCs across the globe, from a central entry point, thus maximizing resource utilization (all QTP and other automation PCs can be configured public and used for testing).

Even though it was originally developed for the test group, the developers now run the automation tests before submitting the s/w to the test engineers. 

SNAP was presented at the SAP world conference last year, and it caused a lot of buzz, in particular raising the motivation and the positioning of the QA engineers in the organization.

Asaf, together with Guy Peled of Gizmox , also presented “Web on the Server”:  at AjaxWorld 2008 East. 

Asaf mentioned that they had fantastic cooperation with HP/Mercury (it doesn’t do any harm if your “daddy” is SAP!) – which helped them get as many licenses as they needed.

You can see a detailed description and screenshots from the Gizmox site by clicking on the screenshots below.

sap1snap_3

Altogether are very interesting and educational presentation.

AutoIT – A Free Functional Automation Tool

The next presentation was given by Meir Bar-Tal, of SOLMAR Knowledge Networks. Meir gave an in-depth talk and demo of AutoIT v3, a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. It uses a combination of simulated keystrokes, mouse movement and window/control manipulation in order to automate tasks in a way not possible or reliable with other languages (e.g. VBScript and SendKeys). AutoIt is also very small, self-contained and will run on all versions of Windows (including Vista) out-of-the-box with no annoying “runtimes” required! 

The evolution of AutoIt

  • Initially designd for PC “roll out” situations to reliably automate and configure 1000’s of PCs
  • Now – used as a general purpose scripting language
  • Now – supports complex expressions, user functions, loops etc.

The main features of AutoIt

  • Easy to learn, Basic-like syntax
  • Simulate keystrokes and mouse movements
  • Manipulate windows and processes
  • Interacts with all standard widow controls
  • Scripts can be compiles into standalone execs
  • Create GUIs
  • COM support
  • Regular expressions
  • Directly call external DLL and Windows API functions
  • Scriptable RunAs functions
  • Detailed Help file with examples, and large community-based support forums
  • Unicode and x64 support
  • Digitally signed for peace of mind
  • Works with Windows Vista’s User Account Control (UAC)

Below are a couple of AutoIT screenshots, so you can get an idea what it looks like.

AutoIT gui_eg1
AutoIT gui_eg2AutoIt also has a community, run by Jonathan Bennett (just a kid – born in 1973) – author and copyright owner, an open community of contributors who are highly prolific and very dedicated. They release a new version every now and again – latest one was on the 24th December 2008. AutoIT has more than 26K forum users worldwide, who are very busy and active, and 2 books have been written about AutoIT.

In conclusion, AutoIT is a cost-effective solution for Web, .NET and Standard windows applications, it is limited to main windows technologies, and is best used with a proven testing automation methodology and framework. AutoIT enables maximizing ROI on automation by effective use of enterprise resources (parallel execution).

And interestingly enough, one of the other attendees at the evening said that he had heard of AutoIT a couple of years ago, but didn’t think it was much use for him and his team. After hearing Meir’s talk, he’s going to have another look.  

AllPairs and PICT – Free tools for test design optimization

Last but by no means least, the third talk was given by Michael Stahl, Senior SW Test Engineer at Intel (although Michael was careful to say that the talk actually has nothing to do with his work at Intel), and member of the ITCB Executive Board.

Michael started off by showing us the following video about the ESP (electronic stability przogram) in a car, which “Enhances driver control andhelps maintain direcctional stabiloity under all conditions. Provides the greatest benefit during critical driving situations, such as when driving on mixed surface conditions such as snow, ice or gravel.”:

[sorry all, I can’t get the stupid video to embed here, so you’ll have to click on the link below to see it]

Exotic-cars.info20.com – ESP – Electronic Stability Program

He then went on to list just a few of the different criteria that need to be considered in order to test an ESP system:

  •  Car Weight
    • Net weight, maximum load, overweight
    • Location of the weight: front seat, back seat, trunk, roof
  • Tires
    • Type, width, size
    • Condition – worn or new
    • Air pressure – nominal, too much, too little
  • Driving conditions
    • Surface – asphalt, concrete, losse gravel, ice, snow, wet, flooded, wet + leaves, oil, sand, etc.!
    • Incline – up, down, flat
    • Contour – left turn, right turn, straight
    • Speed – slow, fast
    • Wind – from left, right, front, back
  • Actions
    • Quiescent, brake, accelerate, down-shift, up-shift

Just combining these criteria, we get to the impossible number of 1,140,480 different test cases !! And that’s just for one car model….

This problem is known as a Combinatorical Explosion. which leads to a “Test Explosion” – more tests than you can ever (want to) run. The problem is very common in software, where you have many configuration parameters, external events, user inputs, environmental parameters, etc.

Ah… I can hear your brains clicking over. You are thinking (as Michael indeed mentioned) “my application doesn’t have this problem”. Either:

  • the parameters are orthoganol (i.e. don’t affect each other), we can test each one on its own
  • the combinations can be covered while testing other things (but that actually makes it more complex)
  • we have enough planned iterations to cover all the combinations (can you be sure that you will be able to control combinations per iteration?)

Uh-uh. Nothing is quite so simple – you need to evaluate the risk of  using any of these strategies.

The other strategy is based on a theory by Tatsumi, and Cohen et al, which states that (bad) interaction between variables is usually between two variables, and that bugs involving interactions between three or more parameters are progressively less common. Therefore, we need to test all of the “all-pair” interactions, and add a few selected tests for specific cases.

Michael then moved on to demo two free tools:

  1. Allpairs  (developed by James Bach)
  2. PICT – Pairwise Independent Combinatorial Testing  (from Microsoft)

He used as an example, a “Generic Installer” with 9 different on/off configuration options, i.e. 512 test cases (2 to the power of 9).

First he demo’d All-Pairs, which gave an answer that just 8 test cases were needed to test all the pairs (instead of 512, remember?)

Next he demo’d PICT, which gave an answer of 9 test cases (also miles better than 512).

Well, this was where All-Pairs stops. It’s a great little free tool, but even James Bach agrees that it has its limitations, the main ones being that there is no logic, and it works only with pairs.

PICT on the other hand, is a bit more complicated, but much more powerful. And it’s also free 🙂   Pict has a pile of options, that help you do any of the following:

  • combination order control
  • randomization
  • constraints ,e.g. if parameter x is “off”, then set parameter y to be “off”
  • sub-models: bundle certain parameters into groups, e.g. if there are some cases you always want to test
  • aliasing: when certain parameter(s) are “don’t care”, i.e. they can take a few values. It doesn’t matter which you use, but it would be nice to test all of them
  • weighting: bias the “don’t care” value distribution to test more important values more than others, e.g. test Vista more than XP
  • negative tests
  • seeding

and it has a very good help file, so go look if there’s anything here that isn’t clear 🙂

In addition, PICT can work with more than 2 parameters. As a real-live example, Michael showed a case in which there were 6 parameters, with between 2 and 22 values per parameter:

A: 2 values, B: 5 values, C: 6, D:7, E:8 and F:22. 

The number of tests ended up being:  2 x 5 x 6 x 7 x 8 x 22 = 73,920.

In the PICT file, 1 constraint was defined, and 56 “if-then-else” equations. All pairs were checked.

The final result (i.e. number of test cases needed, instead of 73,920)???  90. That’s right, NINETY. That’s it!

OK, so this sounds to good to be true. Where’s the catch?  Well, no catch, but some risks:

  • N-pairs is just another tool in your toolbox
    • It’s not guaranteed to find all the bugs
  • The test quality is dependant on the input values you chose
    • Was your Equivalence Class correct>
    • Did you select the “right” representative values?
  • If the output is influenced by more than 2 variables, you need a higher level than all-pairs to catch the bugs
  • Blind selection of paris will miss the “interesting” or often-used combinations.
  • Bach and Shroeder both say: “We believe that this technique is over promoted and poorly understood”

And not to be forgotten, in fact about ALL your tools: ” Don’t fall in love with your tool – it’s not a Silver Bullet. Apply your tester’s instiscts, and analyze the situation”.

I hope you all enjoyed this summary, and can take away something to start doing tomorrow. It would be great if you would add a comment to this post as to what testing improvement using tools YOU are going to start implementing.

And don’t forget, mark the 29th June-2nd July in your calendars, because yours truly (yes, that’s ME) will also be at SIGiST Israel 2009, giving a talk on “Test Engineers – Adding Value in Tough Economic Times“.

Until then, see you at my next post  (if you sign up for my RSS feed, of course!)

debi-zyx-signature

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to FurlAdd to Newsvine

The Best of the Best, SIGiST Israel 2008

And now, last but absolutely not least, the “Best Presentation Award“, which went to Danny (Danko) Kovatch. Danny’s talk was a double session, and it could easily have been a 1 or 2-day tutorial. His subject was “Scrum: the Roles and Responsibilities of QA”, and included a short introduction on Agile and the Agile Manifesto, to get everyone onto the same page.   Danko Best Presentation at SIGiST Israel 2008

Danko used techniques from the world of Agile to explain how testing and testers are involved in Scrum, and his use of the audience was highly effective. Among other things, he taught us how to do “planning poker” – a really cool method of estimating, that gets all the stakeholders in the project actually talking to each other (who’ld have thought it was possible!!) .

Danko explained the dynamics of the Sprint Life Cycle, and showed us a few of the other tools and techniques used in Sprint, like “Planning – just before committing”, Visibility Charts – Task Charts and Burn Down charts.

There’s no way I can explain the dynamics of this session in a few blog lines. Suffice to say that it was highly entertaining as well as very educational.

And if you are asking what the roles and responsibilities of QA are in the world of Scrum?

The role of QA in scrum
 

Now wouldn’t it be nice if we could all say that we work in a fully integrated Scrum organization, where “testers and coders are fully Integrated“??

If you enjoyed this post, please consider subscribing to my RSS Feed, to keep up with all future updates

Add to FacebookAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to Ma.gnoliaAdd to TechnoratiAdd to FurlAdd to Newsvine

Test Management Track (iv), SIGiST Israel 2008

OK, so we come to the last presentation in the Test Management track for the day. There are actually two more test management presentations the following day, but I’m giving my own presentation in the Testing Methodology track, so I’ll be missing them.

Mieke Gevers, from aQis in Belgium, is our last speaker.

Mieke lives and breathes Testing, and in fact she told me that she is “passionate about testing – it is like blood running into her veins”. In addition to her 20 years’ experience as a software developer, analyst, project manager and ultimately QA Manager for several companies in Europe, including Borland and Segue, she is a regular conference speaker throughout Europe, the United States, Asia and Australia ( I really do envy her!). Mieke is also EuroSTAR‘s country coordinator for Belgium, and was a team member of the EuroSTAR 2007 committee.   Mieke Gevers

Continue reading

Test Management Track (iii), SIGiST Israel 2008

3rd Session: Dr. Avi Ofer, Intel, on “The Battle for the Mouse: the Art of ATP“.

Accompanied by some wonderful photographs from his ATP (acceptance test plan) travels around the world, to India, Taiwan and Italy, Avi explained why getting formal acceptance from the customer is probably the most important milestone in the life cycle of a software project – duuuhhh, it’s the stage after which you get paid !!

Avi started with an example of how an inexperienced test engineer managed to insult a Chinese customer, so that Avi had to fly out to China, just after returning from California…

According to Avi, there are 3 types of customer, and any of them can turn up in any part of the world:

  1. The customer who understands Quality and QA, and who is very organized, and has his own SQA (Software Quality Assurance) staff or test team. When you have a customer like this, very often all you need to do is to accompany them, and answer questions – the actual work of the acceptance test is done by the customer’s quality staff. It’s generally a win/win situation. You are there to handle small issues, if they arise. You are there to channel any problems they find directly back to your company, so that they can start fixing them. And you are there so that your CEO knows about any issues before your customer’s CEO does… so that he can be ready with answers if necessary.
  2. The second type of customer doesn’t understand anything about QA. You need to take charge, and run the tests yourself. Make sure that your customer cannot easy get hold of your mouse – don’t let him win the battle – or he may do something to crash the system! On the other hand, you do need to exercise the system as your customer would do, and attempt to find the types of problems that your customer would find. Just make sure that YOU find them.
  3. The final type of customer is a novice. You have to use the opportunity of running the ATP to train him in what the system does. Just be careful not to assume that he’s an idiot just because he doesn’t understand the system. Don’t con him in any way.

Avi explained that in the Far East (especially), the customer can appear as a Fire-breathing Dragon. However, if you take a real interest in his country and culture, he doesn’t have to be.

In any case, this is NOT a task for amateurs: one needs specific skills and experience in various domains, including a deep knowledge of the software life cycle and quality, a customer-oriented approach, a broad cultural horizon for offshore missions, and high flexibility toward the changing conditions during the process.

This activity is commonly (and somewhat wrongly) known as “performing ATP”. But, more than a technical challenge, performing successful ATP is a complicated art.

Test Management Track (ii), SIGiST Israel 2008

OK, I’ve caught up on sleep after 2 very busy days at SIGiST 2008. And looking at my blog posts till now, I think that I’m going to do a few shorter ones for a bit. Here’s the next session from Tuesday’s Test Management track.

2nd Session: Katy Podolner-Rotman, Aternity, on “Surviving “crunch time”: Dealing with constant change, high workload and communication issues“.

Katy started off by giving us her list of “unwritten rules” about how to deal with these issues at crunch time:

DO:

  • Plan for tired people
  • Make time for rest/fun
  • Appreciate people’s work and effort
  • Improve “while running”
  • Communicate – within the team and between teams
  • Laugh 🙂

DONTs:

  • Over schedule
  • Multitask
  • Overwork yourself and others
  • Let small issues develop into huge ones

I like these rules.

Continue reading

Test Management Track (i), SIGiST Israel 2008

1st Session: Yaron Tsubery, Comverse, on “The Psychology Behind Test Management”

Yaron started off the morning with a presentation that can be summed up in three words: “Communication, Communication, Communication“.

Yaron started off with 4 messages that he then elaborated upon during the presentation:

Yaron's messages

After that, he moved on to discuss the need for expectation management between the test manager and all his/her colleagues, and that the test manager has to be proactive in explaining what the test team does to each stakeholder, and getting their feedback.

We were told to view each colleague as a customer, not a new concept, but one that never really took hold in the hi-tech field.

Yaron went on to discuss how easy it is to break trust, as opposed to the effort it takes to build trust.

Transactional Analysis

Next we learnt a bit about Transactional Analysis, a psychoanalytic theory of psychology developed by psychiatrist Eric Berne. Berne postulated three ego states, the Parent, Adult and Child states. Berne considered how individuals interact with one another, and how the ego states affected each set of transactions. This model helps understanding how people function.   Picture of Dr. Eric Berne, author of Games People Play and Originator of Transactional Analysis

Yaron explained Berne’s Interpersonal Conversation Model, where the

Parent (“exteropsychic”) is a state in which people behave, feel and think in response to an unconscious mimicking of how their parents, or other parental figures, acted.

Adult (“neopsychic”) is a state in which people behave, feel and think in response to what is going on in the “here and now,” using all of their resources as an adult human being with many years of life experience to guide them. While a person is in the Adult ego state, he/she is directed towards an objective appraisal of reality.

Child (“archeopsychic”) is a state in which people revert to behaving, feeling and thinking close to how they did in childhood.

There are 2 types of conversational transaction:

  • Reciprocal or Complimentary transactions:  A simple, reciprocal transaction occurs when both partners are addressing the ego-state that the other is in
  • Crossed transactions:  Communication failures are typically caused by a “crossed transaction”, where partners address ego-states other than that which their partner is in

At the end, Yaron gave some tips for prevention, correction, maintenance and some advice for the road, and he closed with his 5th message,

“Remember: It is not enough being a professional to earn your colleagues’ trust, you need also to communicate it in order to earn it!”

More on track day I after I get some sleep …

Intro to 1st Track day at SIGiST Israel 2008

Hi there again,

Today was the first day of the track sessions at the SIGiST Israel 2008 conference. I was honoured to be Track Chair for the Test Management track, which meant that I didn’t have to break my head over which presentations to go and listen to!

The day started with an introduction by Alon Linetzki, the SIGiST Israel Chair and founder, and the guy behind the fact that SIGiST Israel exists – well done, Alon!
Some interesting facts that Alon presented were: more than 90 people attended the tutorials, and more than 140 people were registered for the two track days.
  Alon Linetzki
SIGiST Israel 2008   From having sat through a complete track, I can safely say that these numbers are not an exaggeration. One of the presentations in the management track (by Dr. Avi Ofer) was attended by about 60 people, and most of the others also had almost full-room attendance.
Here are a couple of people trying to decide which sessions to go to 🙂
Alon also mentioned that the number of SIGiST members had grown by 20% in the last year, and that one of the European (I think) SIGiST chairmen had mentioned to him that they reached an attendance of 250 by their 8th conference, so the attendance here was really good, considering that it’s only the 2nd SIGiST Israel conference.

Vipul’s Unfinished Dream

After the opening, the first Keynote speaker of the conference was Vipul Kocher , co-founder of Puretesting and President of the Indian Testing Board, who spoke about his vision for the future of Testing. He ingratiated himself with the audience by starting off with “Shalom”, but then carried on by saying “Layla Tov” (which means “good night”).   Vipul Kocher

He asked us if he had made a mistake, but obviously knew what he was saying, as he then said that he was going to present us his “unfinished dream”, and that one can only dream at night … which was prompted by applause for the play on words in a foreign language. Vipul received another round of applause by saying that if any country had the creativeness to complete his dream it was Israel. 

I did like Vipul’s question to the audience about whether “self-testing software” would solve the problems of testing. I liked his answer even more: “hopefully not in my lifetime!”. For those of us (and that’s probably almost everyone reading this blog) who make our living testing software, I think we can really relate!

Vipul had a wonderful slide in his presentation, in which he said:

Bug – The Only Perfect Being
Most bugs are because
imperfect requirements are imperfectly translated
into imperfect design which is imperfectly translated
into imperfect code

Vipul’s dream itself is an attempt to find breakthrough innovation in testing, and is based on a solution integrating everything that affects the world of testing. This led to a short discussion on whether parts of his solution already exist, and whether it would become a white elephant. Here are a couple of slides to give you an idea of Vipul’s dream.

After the keynote, there were 3 parallel sessions, divided into 4 tracks: Test Management (full day), Test Design and Techniques (full day), Test Methodology (half day) and Test Process Improvement (half day).

More on that to come …