Quest® Benchmark Factory for Databases
Version 6.9.2
Release Notes
August 23, 2013
Welcome to Benchmark Factory for Databases
Quest Software is pleased to make available Benchmark Factory® for Databases 6.9.2 to meet your database workload testing needs. Benchmark Factory is a database performance and code scalability testing tool that simulates users executing transactions on the database and replays production workloads in non-production environments. This enables developers, DBAs, and QA teams to validate database scalability as their user load increases, application changes are made, and platform changes are implemented. Benchmark Factory is available for Oracle, SQL Server, DB2, Sybase, MySQL, and other databases via ODBC connectivity. Benchmark Factory:
Upon completion of a test execution, all test results are collected and stored in the repository for data analysis and reporting. Benchmark Factory collects a variety of informative and detailed statistics that include:
This release of Benchmark Factory includes the following new features and enhancements.
Oracle Capture/Replay Wizard:
Changed the wizards to allow user to select objects to monitor verses the user.
Allow UNC entry for server and client directory.
Changed trace file processing to reduce memory usage.
Greatly decreased the time it takes to process and execute replay of large trace captures.
SQL Server Capture/Replay Wizard:
Change the wizards to allow user to select objects to monitor vs. of user
Allow UNC entry for server and client directory.
Greatly decreased time to execute replay of large trace captures.
Standard Benchmarks:
Added check to verify set scale factor compared with loaded scale factor to benchmark tests.
Added ability to adjust loaded benchmark data to a new scale factor without total recreate/reload.
Support:
Improved support bundle creation.
Add ability to email support bundle.
Added additional information to error output.
Benchmark Factory for Databases now supports Windows 8 and Windows 2012.
The following is a list of issues addressed and enhancements implemented in this release of Benchmark Factory for Databases.
| Feature | Resolved Issue | Defect ID |
|---|---|---|
| Capture and Replay | The following issue has been resolved: After capturing six or more SQL Server workloads, an error was encountered when attempting to capture an additional workload. | 109476 |
| Capture and Replay | Oracle: During an Oracle workload capture using the FGA method, Benchmark Factory saves data to the FGA_LOG$ table (located in the SYSTEM tablespace by default). When a large amount of data is saved, this can cause issues if the tablespace becomes full. This issue in Benchmark Factory has been resolved. | 110992 |
| Integration with SQL Optimizer for Oracle | The following issue has been resolved: The integration between SQL Optimizer for Oracle version 8.8.0 and Benchmark Factory fails when the SQL sent to Benchmark Factory contains single quotes. | 111520 |
| ODBC Drivers | When an ODBC driver name exceeds 255 characters, opening the Repository page of the Benchmark Factory Settings dialog or initiating a support bundle no longer causes an exception error. | 111359 |
| Search Toad World | When using the Search box to search Toad World, results are returned as expected and you no longer encounter a "Sorry, there was a problem with your last request!" error message. | 111333 |
| TPC-C Standard Benchmark | The following issue has been resolved: An "ORA-01405: fetched column value is NULL" error was encountered when running a TPC-C standard benchmark test in Oracle using a scale factor of 4700 on a userload of 200 with a latency reduced to 1/10th of default. | 110994 |
| Feature | Resolved Issue | Defect ID |
|---|---|---|
| Agent | Remove Help button from Agent Settings dialog. | 103959 |
| Agent | Agent error encountered when using ODBC to connect to DB2 database to run a TPC-C test. | 106580 |
| Capture and Replay | When checking the directory, warning messages should display in red text. | 102681 |
| Capture and Replay | Oracle: Removed the "Waiting for connection to be released" message that displayed after clicking the Hide button. | 103966 |
| Capture and Replay | Bind variable error encountered during replay. | 106930 |
| Capture and Replay | The predefined Capture Filter for excluding users with DBA role excludes all with DBA role except the user being watched. | 106931 |
| Capture and Replay | When importing a trace file that has a "Trace file full" message at the end, Benchmark Factory should error out of import and display a similar message. | 107319 |
| Capture and Replay | For an Oracle Capture and Replay using trace files, Benchmark Factory should set the default trace file size parameter to unlimited. | 107320 |
| Capture and Replay | If the capture directory has a space in it, the Statspack Report does not work. | 107339 |
| Capture and Replay | When doing an Oracle Capture/Replay wizard against a 9i database which does not have the StatsPack objects, when submitting the capture an error displays about invalid object (Statspack.Snap) even if you did not selected to do any Statspack reporting. | 107533 |
| General | Sybase: When using the Sybase Client version 15.7 and the Sybase server version 15.0.3, when loading a table for standard, an error similar to "CT-LIBRARY Client Message #4 - blk_bind(): blk layer: user error: Parameter column has an illegal value of 1" displays. | 96878 |
| Integration with SQL Optimizer | When using Benchmark Factory from SQL Optimizer to test SQL for scalability, in the SQL Optimizer Scalability Testing window, when the status is complete, the message in the Message tab does not refresh. | 106306 |
| Load Scenario Wizard | When importing from a delimited text file, now a note in the UI informs the user that only the first 20 rows of the file are displayed in the wizard. | 102562 |
| Load Scenario Wizard | In Replay from Oracle trace file, if the trace file does not contain any valid SQL, clicking Next on the Oracle Trace Activity page results in no action and there is no message. | 106880 |
| Load Scenario Wizard | Benchmark Factory unexpectedly quits when running a TPC-E test in Windows 2008 64-bit or Windows Server 2012 64-bit. | 107493 |
| Load Scenario Wizard | When executing a stored procedure for SQL Server generated by the wizard, the syntax, <database name>.<schema name>.<proc name>, is not followed. | 107492 |
| Profile | The Microsoft SQL Server (ODBC) profile should not require a username since the ODBC DSN may be setup with Windows Authentication. | 106714 |
| Repository | Repository error when the computer name exceeds 20 characters. | 105823 |
| Run Reports | If there is only one report in the list, the Status drop-down list does not contain the option Running. | 103807 |
| Run Reports | In Show Test Results | More Graphs, graphs do not refresh when selecting/comparing more than one test. | 106739 |
| Run Reports | An issue that caused Run Reports to close unexpectedly when importing saved repository results into Benchmark Factory 6.9 from Benchmark Factory 6.8 has been resolved. | 106996 |
| Session Editor | Database information is missing in the SQL Preview window if the profile was created for an Oracle Direct connection instead of TNS. | 103855 |
| Feature | Resolved Issue | Defect ID |
|---|---|---|
| Agents | When doing a benchmark load using Agent(s), and specifying a virtual user count of more than 20 VUs, Benchmark Factory ran locally instead of using the Agent(s). | 106373 |
| Profiles | Error when creating a Profile and checking the Perform Statspack snapshot at the start and end of each test iteration. | 106175 |
| Transaction Wizard | Transaction Wizard > Import Transaction page > Select All button did not work. | 104992 |
| Agents | Benchmark Factory agent maximum users default to zero or unlimited. | 104714 |
| Run Jobs | When choosing Run Job on a completed job, the job moved back up to running and did not change to run job. | 104705 |
| Test Results Wizard | SQL Optimizer Testing Results Window's icon did not update. | 104597 |
| Run Reports | Error when exporting the Run Reports to the network path. | 104358 |
| SQL Preview | During SQL Preview, if the SQL statement had multiple result sets and there was an error running one of the sets, an error message did not display. | 104145 |
| Oracle Trace Files | You can now the import and store of Oracle trace files and store the information in a database/repository to prevent running out of memory. | 103574 |
| Run Reports | Received and error when exporting the Run Reports to network path. | 104597 |
| Check Button | Check button did not refresh the input UNC Path. | 105026 |
| Holding Tables | Incorrect value in holding tables. | 104718 |
| Confirmation Window | Confirmation window did not display correct button. | 104688 |
| Disclosure Report | Incorrect results displayed in the Disclosure Report. | 104320 |
| TPC-E Error | When using the ODBC driver to run the TPC-E benchmark, the TPC-E transaction received an error. | 104076 |
| Excel Reports | When doing an Excel report of a Replay test , the General Tab incorrectly displayed the Mix Workload test showing Sampling and Pre-sampling time. | 104052 |
| Support Bundle | Support bundle did not include the console error log. | 103908 |
| Opening Large SQL Files | Benchmark Factory closes when opening large SQL files. | 103440 |
| Agent Window | Corrected wrong text. | 103347 |
| Testing Replay Load | Benchmark Factory closed when clicking the Back Button during SQL Server trace table analysis. | 103272 |
| AWR Comparison Report | AWR Comparison Report showed improper percentage. | 103201 |
| Replay Performance Reporting | Replay the approximate the performance as that received during capture/. | 103107 |
| SQL Script | The amount of SQL in a script was not equal to that in the guide page of "odbc trace processing" | 103098 |
| Error Message | Required an error message and not a Support Bundle when the database did not have enough storage space. | 102227 |
| Test Connection Pop-Up Message | Test connection pop-up message was misleading and now displays the actual profile user name. | 79474 |
| Error Logging | Add additional information about output was added to error logging. | 38225 |
The following is a list of issues known to exist at the time of this release.
| Feature | Known Issue | Defect ID |
|---|---|---|
| Capture and Replay |
Benchmark Factory uses a .bat file to restore the database while replaying a captured workload. The .bat script does not support a UNC path. Workaround: Map the UNC path to a lettered drive if selecting to restore database when replaying captured workload. |
108839 |
| Capture and Replay |
SQL Server workload: After capturing activity from a user logged-in using Windows authentication, attempting to replay the capture using the ODBC profile results in the following error: "Could not obtain information about Windows NT group/user '<group/user>', error code 0x5." Workaround: Use a domain account to start SQL Server service and make sure the account can connect to the Active Directory of Domain. |
105366 |
| Capture and Replay |
Oracle workload: In an Oracle capture and replay, Benchmark Factory cannot generate the Statspack report if the local replay directory is mapped to drive "B." This is because Benchmark Factory must have drive B open to execute a background .bat file during oracle capture and replay. Workaround: Do not map your local replay directory to drive "B" when running an Oracle capture and replay in which you chose to generate the Statspack report. |
109221 |
| Capture and Replay | In an Oracle 12c container database (CDB), Benchmark Factory cannot replay an existing workload that was captured in an earlier version of an Oracle database. In an Oracle 12c container database, a user name (common user) is prefixed with "c##" by default. | 110936 |
| Capture and Replay |
In Oracle Database 12c, attempting to capture workloads in two (or more) different pluggable databases (PDB) within the same container database (CDB) at the same time is not recommended. If you attempt to do so, the DIAGNOSTIC_DEST parameter might not be restored to the original directory, and the trace files might not be visible in the capture directory. In a related issue, if you close the Capture Scenario Status dialog after scheduling a workload export/capture from a pluggable database (PDB), the workload might not be captured as expected. In addition, while an export/capture is in process, the Hide button in this dialog might be disabled. |
111825 |
| Oracle Database Privileges | In Oracle 12c, the RESOURCE role no longer includes the UNLIMITED TABLESPACE system privilege. Therefore, Benchmark Factory users with the RESOURCE role must also be granted the UNLIMITED TABLESPACE privilege. | 111400 |
| Integration with Foglight Performance Analysis for SQL Server |
When opening Benchmark Factory for scalability testing from Foglight Performance Analysis for SQL Server, "\DEFAULT_INSTANCE" is added to the ODBC DSN. Workaround: On the Connection Information page, click Edit DSN. Remove "\DEFAULT_INSTANCE" from the DSN. |
109137 |
|
Transaction Wizard and IBM CLI Driver
|
When using the 64-bit version of Benchmark Factory, the Transaction Wizard stops working when using the IBM DB2 CLI Driver profile to connect the database which created objects using the ODBC Driver. | 105514 |
| Microsoft Oracle ODBC Driver | When using Microsoft’s Oracle ODBC driver for either testing or for Benchmark Factory repository access, there is an issue with the ODBC driver. It does not work with CLOB data. | N/A |
| Oracle Capture Using FGA | When doing an Oracle Capture using the FGA method, if the application has a trigger with a multi-schema select additional transactions will be recorded. | N/A |
| Oracle ODBC driver |
When using the Oracle 11g R2 Client and Oracle 32-bit ODBC driver in a Windows 64-bit environment, the Oracle 32-bit ODBC driver requires an 8-byte int data type. Workaround: Use the Oracle native client or Oracle 32-bit ODBC driver. |
98249 |
| Licensing | If your current Benchmark Factory license key is a 21-digit format (x-xxxxx-xxxxx-xxxxx-xxxxx) and you want to use the 64-Bit installation, you will need to request a new license key. Please obtain the new key before installing the new release. To obtain a new key, please refer to the Licensing Assistance page on our support portal: Licensing Assistance. | N/A |
| Benchmark Factory Console | Benchmark Factory does not support data types that are created by the user. | N/A |
Before installing Benchmark for Databases, ensure your system meets the following minimum hardware and software requirements:
| Repository |
Benchmark Factory cannot run without a repository, and during installation you must select either the default service based or file based repository. After installation, you can select a repository of your choice. Benchmark Factory supports the following databases:
|
| Memory |
512 MB of RAM recommended Note: The memory required may vary based on the following:
|
| Platform | 1.0 GHz x86 or x64 compatible CPU |
| Browser Requirements | Benchmark Factory for Databases requires IE7 or later. |
| Hard Disk Space | 120 MB of disk space for installation Note: The disk space required may vary based on options selected for installation. |
| Operating System |
Microsoft Windows Server 2003 Microsoft Windows Server 2008 Microsoft XP Professional Microsoft Vista Microsoft Windows 7 Microsoft Windows 8 Microsoft Windows Server 2012 |
| Database Server |
Oracle: 8, 8i, 9i, 9.2, 10g, 11g, and 12c (requires Oracle 8.1.7 client libraries or higher.)
SQL Server: 2000, 2005, 2008, 2008 R2, and 2012 SQL Azure: Benchmark Factory provides limited support of SQL Azure. Sybase: 12.5, 15, and 15.5 DB2 LUW: 8.1.5 through 10.x DB2 for z/OS: 8.x through 10.x MySQL: 4.5 and 5.x
|
| ODBC Database Server | Benchmark Factory for Databases supports almost all databases that you can connect to using an ODBC 3.0 or later driver. |
This section contains information about installing and operating this product in non-English configurations, such as those needed by customers outside of North America. This section does not replace the materials about supported platforms and configurations found elsewhere in the product documentation.
This release is Unicode-enabled and supports any character set. In this release, all product components should be configured to use the same or compatible character encodings and should be installed to use the same locale and regional options. This release is targeted to support operations in the following regions: North America, Western Europe and Latin America, Central and Eastern Europe, Far-East Asia, Japan.
The Benchmark for Databases release package contains the following products:
|
|
|
|
|
Quest Software, Inc. World Headquarters 5 Polaris Way Aliso Viejo, CA 92656 USA |
|
Web site |
See our web site for regional and international office information.
Quest Support is available to customers who have a trial version of a Quest product or who have purchased a Quest product and have a valid maintenance contract. Quest Support provides unlimited 24x7 access to our Support Portal at www.quest.com/support.
From our Support Portal, you can do the following:
View the Global Support Guide for a detailed explanation of support programs, online services, contact information, policies and procedures. The guide is available at: www.quest.com/support.
Get the latest product information, find helpful resources, and join a discussion with the Benchmark Factory team and other community members. Join the Benchmark Factory community at: http://www.toadworld.com/products/benchmark-factory/default.aspx.
© 2013 Quest Software, Inc.
ALL RIGHTS RESERVED.
This guide contains proprietary information protected by copyright. The software described in this guide is furnished under a software license or nondisclosure agreement. This software may be used or copied only in accordance with the terms of the applicable agreement. No part of this guide may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording for any purpose other than the purchaser’s personal use without the written permission of Quest Software, Inc.
The information in this document is provided in connection with Quest products. No license, express or implied, by estoppel or otherwise, to any intellectual property right is granted by this document or in connection with the sale of Quest products. EXCEPT AS SET FORTH IN QUEST'S TERMS AND CONDITIONS AS SPECIFIED IN THE LICENSE AGREEMENT FOR THIS PRODUCT, QUEST ASSUMES NO LIABILITY WHATSOEVER AND DISCLAIMS ANY EXPRESS, IMPLIED OR STATUTORY WARRANTY RELATING TO ITS PRODUCTS INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT. IN NO EVENT SHALL QUEST BE LIABLE FOR ANY DIRECT, INDIRECT, CONSEQUENTIAL, PUNITIVE, SPECIAL OR INCIDENTAL DAMAGES (INCLUDING, WITHOUT LIMITATION, DAMAGES FOR LOSS OF PROFITS, BUSINESS INTERRUPTION OR LOSS OF INFORMATION) ARISING OUT OF THE USE OR INABILITY TO USE THIS DOCUMENT, EVEN IF QUEST HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Quest makes no representations or warranties with respect to the accuracy or completeness of the contents of this document and reserves the right to make changes to specifications and product descriptions at any time without notice. Quest does not make any commitment to update the information contained in this document.
If you have any questions regarding your potential use of this material, contact:
Quest Software World Headquarters
LEGAL Dept
5 Polaris Way
Aliso Viejo, CA 92656
email: legal@quest.com
Refer to our Web site (www.quest.com) for regional and international office information.
Trademarks
Quest, Quest Software, the Quest Software logo, Simplicity at Work, Benchmark Factory, Foglight, Spotlight, SQL Navigator, Toad and T.O.A.D. are trademarks and registered trademarks of Quest Software, Inc. For a complete list of Quest Software’s trademarks, please see http://www.quest.com/legal/trademark-information.aspx. Other trademarks are property of their respective owners.