Benchmarks

 

Benchmarks to Support Evaluation of Concurrent Programs

This website makes available a suite of Java bug-free and faulty benchmarks that was developed to support the validation and evaluation of the structural testing of concurrent programs with message-passing, shared memory and both (message-passing and shared memory) paradigms. Our main focus was to provide a set of programs capable to generate controlled and qualified demand on concurrent software testing artifacts, such as models, criteria and tools.

This suite of benchmarks has a total of 30 bug-free programs that are divided in 13 message-passing, 11 shared memory and 6 with both paradigms. The links below provide the access to the source code and a complete documentation showing the overview, the benchmark objective, expected inputs and outputs, run mode, table with the program features, the pseudo-code and Parallel Control Flow Graph (PCFG) for the benchmarks.

There are also 13 versions of faulty programs, generated from the bug-free benchmarks and used to evaluate the capability of testing tools in reveal defects. Each faulty benchmark contains a realistic bug based on error taxonomies.

____________________                ____________________                ____________________                ____________________
Important warning: to cite the use of these benchmarks in other works/papers, please, use the reference below.

DOURADO, G. G. M.; SOUZA, P. S. L. Contribuindo para a Avaliação do Teste de Programas Concorrentes:
uma abordagem usando benchmarks Master’s thesis, University of São Paulo – ICMC/USP,
São Carlos – SP, Brasil, |2015. In: Portuguese.

By downloading any of those resources, you accept these licenses’ terms and conditions.
Copyright © 2015 Distributed Systemas and Concurrent Programming Laboratory (LaSDPC) –
SSC/ICMC – University of São Paulo (USP). These programs are free software: you can
redistribute it and/or modify it under the terms of the GNU General Public License as
published by the Free Software Foundation, either version 3 of the License, or (at your option)
any later version. These benchmarks are distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program.
If not, see http://www.gnu.org/licenses.
____________________                ____________________                ____________________                ____________________

Downloads

= Bug-Free Benchmarks

    Message-Passing Benchmarks:

      Download: message_passing_benchmarks_code_and_documentation

    Shared Memory Benchmarks:

      Download: shared_memory_benchmarks_code_and_documentation

    Message-Passing and Shared Memory Benchmarks:

      Download: both_paradigms_benchmarks_code_and_documentation

    All Benchmarks Documentation in Portuguese:

      Download: all_benchmarks_documentation_in_portuguese

= Faulty Benchmarks

Download: faulty_benchmarks

Other benchmarks applied to the testing of concurrent programs:

IBM: http://cs.haifa.ac.il/courses/softtest/testing2003/
Rungta: http://wiki.cs.byu.edu/vv-lab/concurrency-tool-comparison/
Valgrind: http://valgrind.org/
Inspect: http://www.cs.utah.edu/~yuyang/inspect/

Contact:

George Gabriel Mendes Dourado (email: georgemd*at*icmc*dot*usp*dot*br)
Paulo Sérgio Lopes de Souza (email: pssouza*at*icmc*dot*usp*dot*br)
LaSDPC/SSC/ICMC/USP