API Testing

Application programmable Interfaces (APIs) are collections of software functions or procedures that can be used by other applications to fulfill their functionality. APIs provide an interface to the software component. These form the critical elements for the developing the applications and are used in varied applications from graph drawing packages, to speech engines, to web-based airline reservation systems, to computer security components.
Each API is supposed to behave the way it is coded, i.e. it is functionality specific. These APIs may offer different results for different type of the input provided. The errors or the exceptions returned may also vary. However once integrated within a product, the common functionality covers a very minimal code path of the API and the functionality testing / integration testing may cover only those paths. By considering each API as a black box, a generalized approach of testing can be applied. But, there may be some paths which are not tested and lead to bugs in the application. Applications can be viewed and treated as APIs from a testing perspective.
There are some distinctive attributes that make testing of APIs slightly different from testing other common software interfaces like GUI testing.

• Testing APIs requires a thorough knowledge of its inner workings - Some APIs may interact with the OS kernel, other APIs, with other software to offer their functionality. Thus an understanding of the inner workings of the interface would help in analyzing the call sequences and detecting the failures caused.
• Adequate programming skills - API tests are generally in the form of sequences of calls, namely, programs. Each tester must possess expertise in the programming language(s) that are targeted by the API. This would help the tester to review and scrutinize the interface under test when the source code is available.
• Lack of Domain knowledge – Since the testers may not be well trained in using the API, a lot of time might be spent in exploring the interfaces and their usage. This problem can be solved to an extent by involving the testers from the initial stage of development. This would help the testers to have some understanding on the interface and avoid exploring while testing.
• No documentation – Experience has shown that it is hard to create precise and readable documentation. The APIs developed will hardly have any proper documentation available. Without the documentation, it is difficult for the test designer to understand the purpose of calls, the parameter types and possible valid/invalid values, their return values, the calls it makes to other functions, and usage scenarios. Hence having proper documentation would help test designer design the tests faster.
• Access to source code – The availability of the source code would help tester to understand and analyze the implementation mechanism used; and can identify the loops or vulnerabilities that may cause errors. Thus if the source code is not available then the tester does not have a chance to find anomalies that may exist in the code.
• Time constraints – Thorough testing of APIs is time consuming, requires a learning overhead and resources to develop tools and design tests. Keeping up with deadlines and ship dates may become a nightmare.

Testing of API calls can be done in isolation or in Sequence to vary the order in which the functionality is exercised and to make the API produce useful results from these tests. Designing tests is essentially designing sequences of API calls that have a potential of satisfying the test objectives. This in turn boils down to designing each call with specific parameters and to building a mechanism for handling and evaluating return values.
Thus designing of the test cases can depend on some of the general questions like

• Which value should a parameter take?
• What values together make sense?
• What combination of parameters will make APIs work in a desired manner?
• What combination will cause a failure, a bad return value, or an anomaly in the operating environment?
• Which sequences are the best candidates for selection? etc.

Some interesting problems for testers being

• Ensuring that the test harness varies parameters of the API calls in ways that verify functionality and expose failures. This includes assigning common parameter values as well as exploring boundary conditions.
• Generating interesting parameter value combinations for calls with two or more parameters.
• Determining the content under which an API call is made. This might include setting external environment conditions (files, peripheral devices, and so forth) and also internal stored data that affect the API.
• Sequencing API calls to vary the order in which the functionality is exercised and to make the API produce useful results from successive calls.

By analyzing the problems listed above, a strategy needs to be formulated for testing the API. The API to be tested would require some environment for it to work. Hence it is required that all the conditions and prerequisites understood by the tester. The next step would be to identify and study its points of entry. The GUIs would have items like menus, buttons, check boxes, and combo lists that would trigger the event or action to be taken. Similarly, for APIs, the input parameters, the events that trigger the API would act as the point of entry. Subsequently, a chief task is to analyze the points of entry as well as significant output items. The input parameters should be tested with the valid and invalid values using strategies like the boundary value analysis and equivalence partitioning. The fourth step is to understand the purpose of the routines, the contexts in which they are to be used. Once all this parameter selections and combinations are designed, different call sequences need to be explored.
The steps can be summarized as following

1. Identify the initial conditions required for testing.
2. Identify the parameters – Choosing the values of individual parameters.
3. Identify the combination of parameters – pick out the possible and applicable parameter combinations with multiple parameters.
4. Identify the order to make the calls – deciding the order in which to make the calls to force the API to exhibit its functionality.
5. Observe the output.

Steps were discussed in details in the forthcoming posts...