Tuesday, March 27, 2012

Analytical Function with examples

This article provides a clear, thorough concept of analytic functions and its various options by a series of simple yet concept building examples. The article is intended for SQL coders, who for might be not be using analytic functions due to unfamiliarity with its cryptic syntax or uncertainty about its logic of operation. Often I see that people tend to reinvent the feature provided by analytic functions by native join and sub-query SQL. This article assumes familiarity with basic Oracle SQL, sub-query, join and group function from the reader. Based on that familiarity, it builds the concept of analytic functions through a series of examples.
It is true that whatever an analytic function does can be done by native SQL, with join and sub-queries. But the same routine done by analytic function is always faster, or at least as fast, when compared to native SQL. Moreover, I am not considering here the amount of time that is spent in coding the native SQLs, testing, debugging and tuning them.
The general syntax of analytic function is:
Function(arg1,..., argn) OVER ( [PARTITION BY <...>] [ORDER BY <....>] [<window_clause>] )
<window_clause> is like "ROW <?>" or "RANK <?>"
All the keywords will be dealt in details as we walk through the examples. The script for creating the schema (SCOTT) on which the example queries of this article are run can be obtained in ORACLE_HOME/sqlplus/demo/demobld.sql of any standard Oracle installation.

How are analytic functions different from group or aggregate functions?



SELECT deptno,
COUNT(*) DEPT_COUNT
FROM emp
WHERE deptno IN (20, 30)
GROUP BY deptno;

DEPTNO                 DEPT_COUNT             
---------------------- ---------------------- 
20                     5                      
30                     6                      

2 rows selected
Query-1

Consider the Query-1 and its result. Query-1 returns departments and their employee count. Most importantly it groups the records into departments in accordance with the GROUP BY clause. As such any non-"group by" column is not allowed in the select clause.

SELECT empno, deptno, 
COUNT(*) OVER (PARTITION BY 
deptno) DEPT_COUNT
FROM emp
WHERE deptno IN (20, 30);

     EMPNO     DEPTNO DEPT_COUNT
---------- ---------- ----------
      7369         20          5
      7566         20          5
      7788         20          5
      7902         20          5
      7876         20          5
      7499         30          6
      7900         30          6
      7844         30          6
      7698         30          6
      7654         30          6
      7521         30          6

11 rows selected.
Query-2

Now consider the analytic function query (Query-2) and its result. Note the repeating values of DEPT_COUNT column.
This brings out the main difference between aggregate and analytic functions. Though analytic functions give aggregate result they do not group the result set. They return the group value multiple times with each record. As such any other non-"group by" column or expression can be present in the select clause, for example, the column EMPNO in Query-2.
Analytic functions are computed after all joins, WHERE clause, GROUP BY and HAVING are computed on the query. The main ORDER BY clause of the query operates after the analytic functions. So analytic functions can only appear in the select list and in the main ORDER BY clause of the query.
In absence of any PARTITION or <window_clause> inside the OVER( ) portion, the function acts on entire record set returned by the where clause. Note the results of Query-3 and compare it with the result of aggregate function query Query-4.

SELECT empno, deptno, 
COUNT(*) OVER ( ) CNT
FROM emp
WHERE deptno IN (10, 20)
ORDER BY 2, 1;

     EMPNO     DEPTNO        CNT
---------- ---------- ----------
      7782         10          8
      7839         10          8
      7934         10          8
      7369         20          8
      7566         20          8
      7788         20          8
      7876         20          8
      7902         20          8
Query-3


SELECT COUNT(*) FROM emp
WHERE deptno IN (10, 20);

  COUNT(*)
----------
         8
Query-4


How to break the result set in groups or partitions?


It might be obvious from the previous example that the clause PARTITION BY is used to break the result set into groups. PARTITION BY can take any non-analytic SQL expression.
Some functions support the <window_clause> inside the partition to further limit the records they act on. In the absence of any <window_clause> analytic functions are computed on all the records of the partition clause.
The functions SUM, COUNT, AVG, MIN, MAX are the common analytic functions the result of which does not depend on the order of the records.
Functions like LEAD, LAG, RANK, DENSE_RANK, ROW_NUMBER, FIRST, FIRST VALUE, LAST, LAST VALUE depends on order of records. In the next example we will see how to specify that.

How to specify the order of the records in the partition?


The answer is simple, by the "ORDER BY" clause inside the OVER( ) clause. This is different from the ORDER BY clause of the main query which comes after WHERE. In this section we go ahead and introduce each of the very useful functions LEAD, LAG, RANK, DENSE_RANK, ROW_NUMBER, FIRST, FIRST VALUE, LAST, LAST VALUE and show how each depend on the order of the record.
The general syntax of specifying the ORDER BY clause in analytic function is:
ORDER BY <sql_expr> [ASC or DESC] NULLS [FIRST or LAST]
The syntax is self-explanatory.

ROW_NUMBER, RANK and DENSE_RANK


All the above three functions assign integer values to the rows depending on their order. That is the reason of clubbing them together.
ROW_NUMBER( ) gives a running serial number to a partition of records. It is very useful in reporting, especially in places where different partitions have their own serial numbers. In Query-5, the function ROW_NUMBER( ) is used to give separate sets of running serial to employees of departments 10 and 20 based on their HIREDATE.

SELECT empno, deptno, hiredate,
ROW_NUMBER( ) OVER (PARTITION BY
deptno ORDER BY hiredate
NULLS LAST) SRLNO
FROM emp
WHERE deptno IN (10, 20)
ORDER BY deptno, SRLNO;

EMPNO  DEPTNO HIREDATE       SRLNO
------ ------- --------- ----------
  7782      10 09-JUN-81          1
  7839      10 17-NOV-81          2
  7934      10 23-JAN-82          3
  7369      20 17-DEC-80          1
  7566      20 02-APR-81          2
  7902      20 03-DEC-81          3
  7788      20 09-DEC-82          4
  7876      20 12-JAN-83          5

8 rows selected.
Query-5 (ROW_NUMBER example)

RANK and DENSE_RANK both provide rank to the records based on some column value or expression. In case of a tie of 2 records at position N, RANK declares 2 positions N and skips position N+1 and gives position N+2 to the next record. While DENSE_RANK declares 2 positions N but does not skip position N+1.
Query-6 shows the usage of both RANK and DENSE_RANK. For DEPTNO 20 there are two contenders for the first position (EMPNO 7788 and 7902). Both RANK and DENSE_RANK declares them as joint toppers. RANK skips the next value that is 2 and next employee EMPNO 7566 is given the position 3. For DENSE_RANK there are no such gaps.

SELECT empno, deptno, sal,
RANK() OVER (PARTITION BY deptno
ORDER BY sal DESC NULLS LAST) RANK,
DENSE_RANK() OVER (PARTITION BY
deptno ORDER BY sal DESC NULLS
LAST) DENSE_RANK
FROM emp
WHERE deptno IN (10, 20)
ORDER BY 2, RANK;

EMPNO  DEPTNO   SAL  RANK DENSE_RANK
------ ------- ----- ----- ----------
  7839      10  5000     1          1
  7782      10  2450     2          2
  7934      10  1300     3          3
  7788      20  3000     1          1
  7902      20  3000     1          1
  7566      20  2975     3          2
  7876      20  1100     4          3
  7369      20   800     5          4

8 rows selected.
Query-6 (RANK and DENSE_RANK example)


LEAD and LAG


LEAD has the ability to compute an expression on the next rows (rows which are going to come after the current row) and return the value to the current row. The general syntax of LEAD is shown below:
LEAD (<sql_expr>, <offset>, <default>) OVER (<analytic_clause>)
<sql_expr> is the expression to compute from the leading row.
<offset> is the index of the leading row relative to the current row.
<offset> is a positive integer with default 1.
<default> is the value to return if the <offset> points to a row outside the partition range.
The syntax of LAG is similar except that the offset for LAG goes into the previous rows.
Query-7 and its result show simple usage of LAG and LEAD function.

SELECT deptno, empno, sal,
LEAD(sal, 1, 0) OVER (PARTITION BY dept ORDER BY sal DESC NULLS LAST) NEXT_LOWER_SAL,
LAG(sal, 1, 0) OVER (PARTITION BY dept ORDER BY sal DESC NULLS LAST) PREV_HIGHER_SAL
FROM emp
WHERE deptno IN (10, 20)
ORDER BY deptno, sal DESC;

 DEPTNO  EMPNO   SAL NEXT_LOWER_SAL PREV_HIGHER_SAL
------- ------ ----- -------------- ---------------
     10   7839  5000           2450               0
     10   7782  2450           1300            5000
     10   7934  1300              0            2450
     20   7788  3000           3000               0
     20   7902  3000           2975            3000
     20   7566  2975           1100            3000
     20   7876  1100            800            2975
     20   7369   800              0            1100

8 rows selected.
Query-7 (LEAD and LAG)


FIRST VALUE and LAST VALUE function


The general syntax is:
FIRST_VALUE(<sql_expr>) OVER (<analytic_clause>)
The FIRST_VALUE analytic function picks the first record from the partition after doing the ORDER BY. The <sql_expr> is computed on the columns of this first record and results are returned. The LAST_VALUE function is used in similar context except that it acts on the last record of the partition.

-- How many days after the first hire of each department were the next
-- employees hired?

SELECT empno, deptno, hiredate ? FIRST_VALUE(hiredate)
OVER (PARTITION BY deptno ORDER BY hiredate) DAY_GAP
FROM emp
WHERE deptno IN (20, 30)
ORDER BY deptno, DAY_GAP;

     EMPNO     DEPTNO    DAY_GAP
---------- ---------- ----------
      7369         20          0
      7566         20        106
      7902         20        351
      7788         20        722
      7876         20        756
      7499         30          0
      7521         30          2
      7698         30         70
      7844         30        200
      7654         30        220
      7900         30        286

11 rows selected.
Query-8 (FIRST_VALUE)


FIRST and LAST function


The FIRST function (or more properly KEEP FIRST function) is used in a very special situation. Suppose we rank a group of record and found several records in the first rank. Now we want to apply an aggregate function on the records of the first rank. KEEP FIRST enables that.
The general syntax is:
Function( ) KEEP (DENSE_RANK FIRST ORDER BY <expr>) OVER (<partitioning_clause>)
Please note that FIRST and LAST are the only functions that deviate from the general syntax of analytic functions. They do not have the ORDER BY inside the OVER clause. Neither do they support any <window> clause. The ranking done in FIRST and LAST is always DENSE_RANK. The query below shows the usage of FIRST function. The LAST function is used in similar context to perform computations on last ranked records.

-- How each employee's salary compare with the average salary of the first
-- year hires of their department?

SELECT empno, deptno, TO_CHAR(hiredate,'YYYY') HIRE_YR, sal,
TRUNC(
AVG(sal) KEEP (DENSE_RANK FIRST
ORDER BY TO_CHAR(hiredate,'YYYY') )
OVER (PARTITION BY deptno)
     ) AVG_SAL_YR1_HIRE
FROM emp
WHERE deptno IN (20, 10)
ORDER BY deptno, empno, HIRE_YR;

     EMPNO     DEPTNO HIRE        SAL AVG_SAL_YR1_HIRE
---------- ---------- ---- ---------- ----------------
      7782         10 1981       2450             3725
      7839         10 1981       5000             3725
      7934         10 1982       1300             3725
      7369         20 1980        800              800
      7566         20 1981       2975              800
      7788         20 1982       3000              800
      7876         20 1983       1100              800
      7902         20 1981       3000              800

8 rows selected.
Query-9 (KEEP FIRST)


How to specify the Window clause (ROW type or RANGE type windows)?


Some analytic functions (AVG, COUNT, FIRST_VALUE, LAST_VALUE, MAX, MIN and SUM among the ones we discussed) can take a window clause to further sub-partition the result and apply the analytic function. An important feature of the windowing clause is that it is dynamic in nature.
The general syntax of the <window_clause> is

[ROW or RANGE] BETWEEN <start_expr> AND <end_expr>
<start_expr> can be any one of the following
  1. UNBOUNDED PECEDING
  2. CURRENT ROW
  3. <sql_expr> PRECEDING or FOLLOWING.

    <end_expr> can be any one of the following
  1. UNBOUNDED FOLLOWING or
  2. CURRENT ROW or
  3. <sql_expr> PRECEDING or FOLLOWING.
For ROW type windows the definition is in terms of row numbers before or after the current row. So for ROW type windows <sql_expr> must evaluate to a positive integer.
For RANGE type windows the definition is in terms of values before or after the current ORDER. We will take this up in details latter.
The ROW or RANGE window cannot appear together in one OVER clause. The window clause is defined in terms of the current row. But may or may not include the current row. The start point of the window and the end point of the window can finish before the current row or after the current row. Only start point cannot come after the end point of the window. In case any point of the window is undefined the default is UNBOUNDED PRECEDING for <start_expr> and UNBOUNDED FOLLOWING for <end_expr>.
If the end point is the current row, syntax only in terms of the start point can be can be
[ROW or RANGE] [<start_expr> PRECEDING or UNBOUNDED PRECEDING ]
[ROW or RANGE] CURRENT ROW is also allowed but this is redundant. In this case the function behaves as a single-row function and acts only on the current row.

ROW Type Windows


For analytic functions with ROW type windows, the general syntax is:
Function( ) OVER (PARTITIN BY <expr1> ORDER BY <expr2,..> ROWS BETWEEN <start_expr> AND <end_expr>)
or
Function( ) OVER (PARTITON BY <expr1> ORDER BY <expr2,..> ROWS [<start_expr> PRECEDING or UNBOUNDED PRECEDING]
For ROW type windows the windowing clause is in terms of record numbers.
The query Query-10 has no apparent real life description (except column FROM_PU_C) but the various windowing clause are illustrated by a COUNT(*) function. The count simply shows the number of rows inside the window definition. Note the build up of the count for each column for the YEAR 1981.
The column FROM_P3_TO_F1 shows an example where start point of the window is before the current row and end point of the window is after current row. This is a 5 row window; it shows values less than 5 during the beginning and end.

-- The query below has no apparent real life description (except 
-- column FROM_PU_C) but is remarkable in illustrating the various windowing
-- clause by a COUNT(*) function.
 
SELECT empno, deptno, TO_CHAR(hiredate, 'YYYY') YEAR,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 3 PRECEDING AND 1 FOLLOWING) FROM_P3_TO_F1,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM_PU_TO_C,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 3 PRECEDING AND 1 PRECEDING) FROM_P2_TO_P1,
COUNT(*) OVER (PARTITION BY TO_CHAR(hiredate, 'YYYY')
ORDER BY hiredate ROWS BETWEEN 1 FOLLOWING AND 3 FOLLOWING) FROM_F1_TO_F3
FROM emp
ORDEDR BY hiredate

 EMPNO  DEPTNO YEAR FROM_P3_TO_F1 FROM_PU_TO_C FROM_P2_TO_P1 FROM_F1_TO_F3
------ ------- ---- ------------- ------------ ------------- -------------
  7369      20 1980             1            1             0             0
  <font bgcolor=yellow>7499      30 1981             2            1             0             3
  7521      30 1981             3            2             1             3
  7566      20 1981             4            3             2             3
  7698      30 1981             5            4             3             3
  7782      10 1981             5            5             3             3
  7844      30 1981             5            6             3             3
  7654      30 1981             5            7             3             3
  7839      10 1981             5            8             3             2
  7900      30 1981             5            9             3             1
  7902      20 1981             4           10             3             0</font>
  7934      10 1982             2            1             0             1
  7788      20 1982             2            2             1             0
  7876      20 1983             1            1             0             0

14 rows selected.
Query-10 (ROW type windowing example)

The column FROM_PU_TO_CURR shows an example where start point of the window is before the current row and end point of the window is the current row. This column only has some real world significance. It can be thought of as the yearly employee build-up of the organization as each employee is getting hired.
The column FROM_P2_TO_P1 shows an example where start point of the window is before the current row and end point of the window is before the current row. This is a 3 row window and the count remains constant after it has got 3 previous rows.
The column FROM_F1_TO_F3 shows an example where start point of the window is after the current row and end point of the window is after the current row. This is a reverse of the previous column. Note how the count declines during the end.

RANGE Windows


For RANGE windows the general syntax is same as that of ROW:
Function( ) OVER (PARTITION BY <expr1> ORDER BY <expr2> RANGE BETWEEN <start_expr> AND <end_expr>)
or
Function( ) OVER (PARTITION BY <expr1> ORDER BY <expr2> RANGE [<start_expr> PRECEDING or UNBOUNDED PRECEDING]

For <start_expr> or <end_expr> we can use UNBOUNDED PECEDING, CURRENT ROW or <sql_expr> PRECEDING or FOLLOWING. However for RANGE type windows <sql_expr> must evaluate to value compatible with ORDER BY expression <expr1>.
<sql_expr> is a logical offset. It must be a constant or expression that evaluates to a positive numeric value or an interval literal. Only one ORDER BY expression is allowed.
If <sql_expr> evaluates to a numeric value, then the ORDER BY expr must be a NUMBER or DATE datatype. If <sql_expr> evaluates to an interval value, then the ORDER BY expr must be a DATE datatype.
Note the example (Query-11) below which uses RANGE windowing. The important thing here is that the size of the window in terms of the number of records can vary.

-- For each employee give the count of employees getting half more that their 
-- salary and also the count of employees in the departments 20 and 30 getting half 
-- less than their salary.
 
SELECT deptno, empno, sal,
Count(*) OVER (PARTITION BY deptno ORDER BY sal RANGE
BETWEEN UNBOUNDED PRECEDING AND (sal/2) PRECEDING) CNT_LT_HALF,
COUNT(*) OVER (PARTITION BY deptno ORDER BY sal RANGE
BETWEEN (sal/2) FOLLOWING AND UNBOUNDED FOLLOWING) CNT_MT_HALF
FROM emp
WHERE deptno IN (20, 30)
ORDER BY deptno, sal

 DEPTNO  EMPNO   SAL CNT_LT_HALF CNT_MT_HALF
------- ------ ----- ----------- -----------
     20   7369   800           0           3
     20   7876  1100           0           3
     20   7566  2975           2           0
     20   7788  3000           2           0
     20   7902  3000           2           0
     30   7900   950           0           3
     30   7521  1250           0           1
     30   7654  1250           0           1
     30   7844  1500           0           1
     30   7499  1600           0           1
     30   7698  2850           3           0

11 rows selected.
Query-11 (RANGE type windowing example)


Order of computation and performance tips


Defining the PARTITOIN BY and ORDER BY clauses on indexed columns (ordered in accordance with the PARTITION CLAUSE and then the ORDER BY clause in analytic function) will provide optimum performance. For Query-5, for example, a composite index on (deptno, hiredate) columns will prove effective.
It is advisable to always use CBO for queries using analytic functions. The tables and indexes should be analyzed and optimizer mode should be CHOOSE.
Even in absence of indexes analytic functions provide acceptable performance but need to do sorting for computing partition and order by clause. If the query contains multiple analytic functions, sorting and partitioning on two different columns should be avoided if they are both not indexed.

Conclusion


The aim of this article is not to make the reader try analytic functions forcibly in every other complex SQL. It is meant for a SQL coder, who has been avoiding analytic functions till now, even in complex analytic queries and reinventing the same feature much painstakingly by native SQL and join query. Its job is done if such a person finds analytic functions clear, understandable and usable after going through the article, and starts using them.

Thursday, March 15, 2012

Oracle WMSYS Functions

conn wmsys/wmsys


WM_CONCAT 
Cross-tabulates a comma delimited list 


WM_CONCAT(p1 IN VARCHAR2) RETURN VARCHAR2 


CREATE TABLE t (
col1 VARCHAR2(5),
col2 VARCHAR2(20));

INSERT INTO t VALUES (111, 'This');
INSERT INTO t VALUES (111, 'is');
INSERT INTO t VALUES (111, 'a');
INSERT INTO t VALUES (111, 'test');
INSERT INTO t VALUES (222, 'This is not');

SELECT * FROM t;

col concat format a40

SELECT col1, wmsys.wm_concat(col2) CONCAT
FROM t
GROUP BY col1;

SELECT col1, TRANSLATE(wmsys.wm_concat(col2), 'A,', 'A ') CONCAT
FROM t
GROUP BY col1; 







WM_CONTAINS 


Checks if the first period contains the second period 


WM_CONTAINS(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN INTEGER; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_CONTAINS';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_CONTAINS') FROM dual;

set serveroutput on

DECLARE
 retval INTEGER;
 p1b    DATE := TO_DATE('01-JAN-2010');
 p1e    DATE := TO_DATE('05-JAN-2010');
 p2b    DATE := TO_DATE('01-JAN-2010');
 p2e    DATE := TO_DATE('03-JAN-2010');
BEGIN
  SELECT WM_CONTAINS(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);

  p1b := TO_DATE('02-JAN-2010');
  p1e := TO_DATE('06-JAN-2010');

  SELECT WM_CONTAINS(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);  
END;
/ 





WM_EQUALS 


Checks if two periods are equal (that is, their start and end times are the same) 


WM_EQUALS(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN INTEGER; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_EQUALS';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_EQUALS') FROM dual;

set serveroutput on

DECLARE
 retval INTEGER;
 p1b    DATE := TO_DATE('01-JAN-2010');
 p1e    DATE := TO_DATE('03-JAN-2010');
 p2b    DATE := TO_DATE('01-JAN-2010');
 p2e    DATE := TO_DATE('03-JAN-2010');
BEGIN
  SELECT WM_EQUALS(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);

  p1b := TO_DATE('01-JAN-2010');
  p1e := TO_DATE('04-JAN-2010');

  SELECT WM_EQUALS(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);  
END; 

/


WM_GREATERTHAN 


Checks if the start of the first period is greater than (that is, later than) the end of the second period 


WM_GREATERTHAN(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN INTEGER; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_GREATERTHAN';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_GREATERTHAN') FROM dual;

set serveroutput on

DECLARE
 retval INTEGER;
 p1b    DATE := TO_DATE('01-FEB-2010');
 p1e    DATE := TO_DATE('03-FEB-2010');
 p2b    DATE := TO_DATE('01-JAN-2010');
 p2e    DATE := TO_DATE('31-JAN-2010');
BEGIN
  SELECT WM_GREATERTHAN(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);

  p2b := TO_DATE('02-FEB-2010');
  p2e := TO_DATE('31-DEC-2010');

  SELECT WM_GREATERTHAN(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);  
END;
/ 



WM_INTERSECTION 


Returns the intersection of the two periods, that is, the time range common to both periods 


WM_INTERSECTION(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN WM_PERIOD; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_INTERSECTION';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_INTERSECTION') FROM dual;

set serveroutput on

DECLARE
 retval WM_PERIOD;
 p1b    DATE := TO_DATE('01-JAN-2010');
 p1e    DATE := TO_DATE('10-JAN-2010');
 p2b    DATE := TO_DATE('08-JAN-2010');
 p2e    DATE := TO_DATE('12-JAN-2010');
BEGIN
  SELECT WM_INTERSECTION(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval.validfrom);
  dbms_output.put_line(retval.validtill);
END;
/ 



WM_LDIFF 


Returns the difference between the two periods on the left (that is, earlier in time) 


WM_LDIFF(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN WM_PERIOD; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_LDIFF';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_LDIFF') FROM dual;

set serveroutput on

DECLARE
 retval WM_PERIOD;
 p1b    DATE := TO_DATE('01-NOV-2010');
 p1e    DATE := TO_DATE('31-DEC-2010');
 p2b    DATE := TO_DATE('29-NOV-2010');
 p2e    DATE := TO_DATE('01-DEC-2010');
BEGIN
  SELECT WM_LDIFF(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval.validfrom);
  dbms_output.put_line(retval.validtill);
END;
/ 



WM_RDIFF 


Returns the difference between the two periods on the right (that is, later in time 


WM_RDIFF(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN WM_PERIOD; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_RDIFF';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_RDIFF') FROM dual;

set serveroutput on

DECLARE
 retval WM_PERIOD;
 p1b    DATE := TO_DATE('01-JAN-2010');
 p1e    DATE := TO_DATE('31-DEC-2010');
 p2b    DATE := TO_DATE('12-JAN-2009');
 p2e    DATE := TO_DATE('16-JAN-2010');
BEGIN
  SELECT WM_RDIFF(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval.validfrom);
  dbms_output.put_line(retval.validtill);
END;
/ 



WM_LESSTHAN 


Checks if the end of the first period is less than (that is, earlier than) the start of the second period 


WM_LESSTHAN(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN INTEGER; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_LESSTHAN';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_LESSTHAN') FROM dual;

set serveroutput on

DECLARE
 retval INTEGER;
 p1b    DATE := TO_DATE('01-JAN-2010');
 p1e    DATE := TO_DATE('02-JAN-2010');
 p2b    DATE := TO_DATE('03-JAN-2010');
 p2e    DATE := TO_DATE('16-JAN-2010');
BEGIN
  SELECT WM_LESSTHAN(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);

  p1b := TO_DATE('04-JAN-2010');
  p1e := TO_DATE('10-JAN-2010');

  SELECT WM_LESSTHAN(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);  
END;
/ 





WM_MEETS 


Checks if the end of the first period is the start of the second period 


WM_MEETS(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN INTEGER; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_MEETS';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_OVERLAPS') FROM dual;

set serveroutput on

DECLARE
 retval INTEGER;
 p1b    DATE := TO_DATE('01-JAN-2010');
 p1e    DATE := TO_DATE('03-JAN-2010');
 p2b    DATE := TO_DATE('03-JAN-2010');
 p2e    DATE := TO_DATE('06-JAN-2010');
BEGIN
  SELECT WM_MEETS(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);

  p1b := TO_DATE('01-JAN-2010');
  p1e := TO_DATE('04-JAN-2010');

  SELECT WM_MEETS(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);  
END;
/ 



WM_OVERLAPS 


Checks if two periods overlap 


WM_OVERLAPS(p1 IN WM_PERIOD, p2 IN WM_PERIOD) RETURN NUMBER; 


SELECT number_of_binds
FROM user_operators
WHERE owner = 'WMSYS'
AND operator_name = 'WM_OVERLAPS';

set long 1000000

SELECT dbms_metadata.get_ddl('OPERATOR', 'WM_OVERLAPS') FROM dual;

set serveroutput on

DECLARE
 retval INTEGER;
 p1b    DATE := TO_DATE('01-JAN-2010');
 p1e    DATE := TO_DATE('31-JAN-2010');
 p2b    DATE := TO_DATE('31-DEC-2009');
 p2e    DATE := TO_DATE('02-JAN-2010');
BEGIN
  SELECT WM_OVERLAPS(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);

  p1b := TO_DATE('03-JAN-2010');
  p1e := TO_DATE('31-JAN-2010');

  SELECT WM_OVERLAPS(WM_PERIOD(p1b, p1e), WM_PERIOD(p2b, p2e))
  INTO retval
  FROM dual;

  dbms_output.put_line(retval);  
END;
/ 






Oracle hidden utilities undocumented parameters

Each new release of Oracle brings new hidden utilities. They are sometimes used by internal Oracle development staff and left in Oracle for emergency support. Many of these undocumented utilities are very powerful but can be complex.
A hidden utility is not a utility that is physically hidden in the Oracle software. Rather, a hidden utility is an executable or PL/SQL package that is either undocumented or where the documentation is difficult to find. For example, some PL/SQL packages are never loaded by the Oracle installer, yet their definitions remain in the operating system files.
The main directories of interest regarding Oracle utilities are the following:
  • $ORACLE_HOME/bin -- This contains the binary executables used by the Oracle server. Most of the tools discussed in this book reside here.
  • $ORACLE_HOME/plsql/demo -- This contains a useful collection of SQL scripts related to many utilities, including dbms_profiler.
  • $ORACLE_HOME/rdbms/admin -- This contains many SQL scripts used for creating PL/SQL packages and their required environments.
  • $ORACLE_HOME/sqlplus/admin -- This contains scripts used with autotrace and other utilities.
  • $ORACLE_HOME/otrace/admin -- This is the administration directory for the Oracle Trace diagnostic tool.
  • $ORACLE_HOME/otrace/sysman -- This is used by utilities such as oemctl and the Oracle Management Server (OMS).
  • $ORACLE_HOME/otrace/lib -- This contains facility files used with the oerr utility.
Many of these undocumented utilities such as TKPROF have surfaced from the obscure and entered mainstream Oracle toolkits while others remain hidden inside the O/S.

Finding hidden utilities

The easiest way to find hidden utilities is to look for new packages within Oracle. The following query compares the packages in Oracle8i with new packages in Oracle9i, using a database link:
select 
   object_name 
from 
   dba_objects@oracle9i 
where 
   owner      = 'SYS' 
and 
   object_type = 'PACKAGE BODY' 
minus
select 
   object_name 
from  
   dba_objects@oracle8i
where 
   owner     = 'SYS'
and 
   object_type = 'PACKAGE BODY';
This query will quickly display all package bodies owned by SYS that exist in Oracle9i but not in Oracle8i.

Finding hidden executables

Discovering new binaries entails comparing two directories in the operating system and ignoring the duplicates entries. The dircmp UNIX command can be used to find only new entries within two directories:

dircmp -- s 
/u01/aoracle/product/8.1.7/bin 
/u01/oracle/product/9.0.3/bin
Here we compare the bin directory of 8.1.7 with Oracle 9.0.2. Note that the -- s option of the dircmp command tells UNIX to eliminate the matches from the output.

Hidden initialization parameters

Hidden initialization parameters are very dangerous because their use is undocumented, but they can be very valuable if you are careful. We can use a SQL*Plus query to quickly find any new initialization parameters in a new version of Oracle:
select 
   name
from 
    v$parameter@oracle9i 
minus 
select 
   name
from 
   v$parameter@oracle8i
; 
Every version of Oracle has special undocumented initialization parameters. These undocumented initialization parameters are usually only used in emergencies and only under the direction of a senior DBA or Oracle support. Because the undocumented parameters begin with an underscore "_", we can write a query against the X$ foxed tables to easily extract them:
COLUMN parameter           FORMAT a37
COLUMN description         FORMAT a30 WORD_WRAPPED
COLUMN "Session Value"     FORMAT a10
COLUMN "Instance Value"    FORMAT a10
SET LINES 100
SET PAGES 0
SPOOL undoc.lis

SELECT  
   a.ksppinm  "Parameter",  
   a.ksppdesc "Description", 
   b.ksppstvl "Session Value",
   c.ksppstvl "Instance Value"
FROM 
   x$ksppi a, 
   x$ksppcv b, 
   x$ksppsv c
WHERE 
   a.indx = b.indx 
   AND 
   a.indx = c.indx
   AND 
   a.ksppinm LIKE '/_%' escape '/'
/
Remember, not everyone knows about the undocumented parameters, and few know how or when to use them. Oracle does not allow DBAs to use many of these parameters unless specifically directed by Oracle support. DBAs should be aware that use of certain undocumented parameters will result in an unsupported system.
In many cases, the undocumented parameters were either documented in previous releases or will be in future releases. Of course, it is difficult to use the undocumented parameters that have never been documented, and never will be, safely. When in doubt, get guidance from Oracle support. We will take a closer look at some of these parameters in our next installment.

Hidden v$ views

New V$ views indicate a new functionality within the database, and not all of the views make it into the Oracle documentation. Here is a quick query to locate new views:
select 
   view_name
from 
    v$fixed_view_description@oracle9i 
minus 
select 
   view_name
from 
   v$fixed_view_description@oracle8i
; 

Hidden system events

With each new release of Oracle, system events and system statistics are changed. For example, here is a query to show changed system statistics for the database writer process between Oracle8i and Oracle9i:
sql> select distinct name from v$sysstat where name like 'DBWR%'

NAME
----------------------------------------------------------------
DBWR Flush object call found no dirty buffers
DBWR Flush object cross instance calls
DBWR buffers scanned
DBWR checkpoint buffers written
DBWR checkpoint write requests
DBWR checkpoints
DBWR cross instance writes
DBWR free buffers found
DBWR incr. ckpt. write requests
DBWR lru scans
DBWR make free requests
DBWR revisited being-written buffer
DBWR skip hot writes
DBWR summed scan depth
DBWR timeouts
DBWR transaction table writes
DBWR undo block writes
Here is the listing from Oracle9i.  Note the changed statistics.
SQL> select distinct name from v$sysstat where name like 'DBWR%';

NAME
----------------------------------------------------------------
DBWR buffers scanned
DBWR checkpoint buffers written
DBWR checkpoints
DBWR cross instance writes
DBWR free buffers found
DBWR fusion writes
DBWR lru scans
DBWR make free requests
DBWR revisited being-written buffer
DBWR summed scan depth
DBWR transaction table writes
DBWR undo block writes
By the same token, we van use the v $event_name view to quickly locate all new system events in Oracle19i.  Here is a quick query to locate new views:
select 
   name
from 
    v$event_name@oracle9i 
minus 
select 
   name
from 
   v$event_name@oracle8i
; 

Undocumented SQL hints

Getting a list of Oracle hints for SQL tuning is often difficult. The Oracle hint list is inside the Oracle executable and you can extract the Oracle hint list easily with UNIX commands. In UNIX, you can use grep and strings to get them directly from the Oracle executable:
   strings $ORACLE_HOME/bin/oracle > hints.lst
Here are all of the 124 Oracle9i hints on release 9.0.2:
ALL_ROWSAND_EQUALANTIJOIN
APPENDBITMAPBUFFER
BYPASS_RECURSIVE_CHECKBYPASS_UJVCCACHE
CACHE_CBCACHE_TEMP_TABLECARDINALITY
CHOOSECIV_GBCOLLECTIONS_GET_REFS
CPU_COSTINGCUBE_GBCURSOR_SHARING_EXACT
DEREF_NO_REWRITEDML_UPDATEDOMAIN_INDEX_NO_SORT
DOMAIN_INDEX_SORTDRIVING_SITEDYNAMIC_SAMPLING
DYNAMIC_SAMPLING_EST_CDNEXPAND_GSET_TO_UNIONFACT
FIRST_ROWSFORCE_SAMPLE_BLOCKFULL
GBY_CONC_ROLLUPGLOBAL_TABLE_HINTSHASH
HASH_AJHASH_SJHWM_BROKERED
IGNORE_ON_CLAUSEIGNORE_WHERE_CLAUSEINDEX_ASC
INDEX_COMBINEINDEX_DESCINDEX_FFS
INDEX_JOININDEX_RRSINDEX_SS
INDEX_SS_ASCINDEX_SS_DESCINLINE
LEADINGLIKE_EXPANDLOCAL_INDEXES
MATERIALIZEMERGEMERGE_AJ
MERGE_SJMV_MERGENESTED_TABLE_GET_REFS
NESTED_TABLE_SET_REFSNESTED_TABLE_SET_SETIDNL_AJ
NL_SJNO_ACCESSNO_BUFFER
NO_EXPANDNO_EXPAND_GSET_TO_UNIONNO_FACT
NO_FILTERINGNO_INDEXNO_MERGE
NO_MONITORINGNO_ORDER_ROLLUPSNO_PRUNE_GSETS
NO_PUSH_PREDNO_PUSH_SUBQNO_QKN_BUFF
NO_SEMIJOINNO_STATS_GSETSNO_UNNEST
NOAPPENDNOCACHENOCPU_COSTING
NOPARALLELNOPARALLEL_INDEXNOREWRITE
OR_EXPANDORDEREDORDERED_PREDICATES
OVERFLOW_NOMOVEPARALLELPARALLEL_INDEX
PIV_GBPIV_SSFPQ_DISTRIBUTE
PQ_MAPPQ_NOMAPPUSH_PRED
PUSH_SUBQREMOTE_MAPPEDRESTORE_AS_INTERVALS
REWRITERULESAVE_AS_INTERVALS
SCN_ASCENDINGSELECTIVITYSEMIJOIN
SEMIJOIN_DRIVERSKIP_EXT_OPTIMIZERSQLLDR
STARSTAR_TRANSFORMATIONSWAP_JOIN_INPUTS
SYS_DL_CURSORSYS_PARALLEL_TXNSYS_RID_ORDER
TIV_GBTIV_SSFUNNEST
USE_ANTIUSE_CONCATUSE_HASH
USE_MERGEUSE_NLUSE_SEMI
USE_TTT_FOR_GSETS
In this output we see 57 undocumented SQL hints:
BYPASS_RECURSIVE_CHECKBYPASS_UJVCCACHE_CB
CACHE_TEMP_TABLECIV_GBCOLLECTIONS_GET_REFS
CUBE_GBCURSOR_SHARING_EXACTDEREF_NO_REWRITE
DML_UPDATEDOMAIN_INDEX_NO_SORTDOMAIN_INDEX_SORT
DYNAMIC_SAMPLINGDYNAMIC_SAMPLING_EST_CDNEXPAND_GSET_TO_UNION
FORCE_SAMPLE_BLOCKGBY_CONC_ROLLUPGLOBAL_TABLE_HINTS
HWM_BROKEREDIGNORE_ON_CLAUSEIGNORE_WHERE_CLAUSE
INDEX_RRSINDEX_SSINDEX_SS_ASC
INDEX_SS_DESCLIKE_EXPANDLOCAL_INDEXES
MV_MERGENESTED_TABLE_GET_REFSNESTED_TABLE_SET_REFS
NESTED_TABLE_SET_SETIDNO_EXPAND_GSET_TO_UNIONNO_FACT
NO_FILTERINGNO_ORDER_ROLLUPSNO_PRUNE_GSETS
NO_STATS_GSETSNO_UNNESTNOCPU_COSTING
OVERFLOW_NOMOVEPIV_GBPIV_SSF
PQ_MAPPQ_NOMAPREMOTE_MAPPED
RESTORE_AS_INTERVALSSAVE_AS_INTERVALSSCN_ASCENDING
SKIP_EXT_OPTIMIZERSQLLDRSYS_DL_CURSOR
SYS_PARALLEL_TXNSYS_RID_ORDERTIV_GB
TIV_SSFUNNESTUSE_TTT_FOR_GSETS
These Oracle SQL hints can be extremely useful for solving complex SQL execution problems, and that is why Oracle created them. They generally remain undocumented because Oracle does not want someone with inadequate knowledge of optimizer internals using them because they could lead to confusing results. However, for the experienced SQL tuning professional these undocumented hints are a godsend.

Conclusion

In this first installment we have shown you how to extract powerful (but potentially dangerous) undocumented utilities, parameters, hints, packages and executables. Remember, many of these undocumented features have unknown effects and should be treated with great care.

Oracle Escape Characters

Oracle allows the assignment of special escape characters to tell Oracle that the character is interpreted literally.  Certain characters ...