ORACLENERD

Afraid to COMMIT;

2010-03-14T18:07:00.005-04:00

Going through some old documents, I found this little gem I had uncovered doing some analysis of the source code.

I found this crazy enough to save, so I hope you enjoy it.

I found 14 or 15 commits in a 115 line procedure. I was shocked and stunned. It was wrong on so many different levels. I share with you the pain I went through.

We'll start off with a call to the logging table (just a quick note, this was not a stored procedure but a INSERT statement).

log_something;

Creating a record in the THIS_TABLE, it will have a status of A. Everything in there has a status of A.

INSERT INTO this_table (my_id, start_date) 
VALUES ( l_my_id, sysdate)
RETURNING table_id INTO l_table_id;

I know some could argue for COMMITs being inside stored procedure, but it was hammered into my head at an early age that the calling application should perform the COMMIT.

COMMIT;

THIS_TABLE had multiple "rules" tables. I understood the concept, but the implementation was not so good. This is the first "rules" table.

UPDATE rule_tab
  SET status = 'D'
WHERE my_id = l_my_id
  AND status = 'A';

Just so you are aware, that STATUS column had no constraint on it other than the size VARCHAR2(1).

Guess what time it is?

COMMIT;

Here is the second

UPDATE other_rules_table
  SET status='D'
WHERE my_id = l_my_id
  AND status='A';

Guess what?

COMMIT;

OK...here comes the other awesome part...to switch the status back to A (Active), we're going to create a job...in 30 minutes

dbms_job.submit
  ( job => ln_jobno, 
    what => 'update rule_table set status=''A'' where my_id = '||l_my_id||' AND status = ''D'';', 
    next => sysdate+(.5/24));

COMMIT;

I kind of understand that one...for the job to go into the queue you have to issue the COMMIT.

For some reason, it's now cool to use a nested block.

BEGIN
  dbms_job.submit
    ( job => ln_jobno, 
      what => 'update other_rule_table set status=''A'' where my_id = '||l_my_id||' AND status = ''D'';', 
      next => sysdate+(.5/24) );

  COMMIT;
EXCEPTION
  WHEN OTHERS THEN
     insert into errors (name,error_date, text) 
     values ('it broke', sysdate, ' MY_ID '||l_my_id||' creating job  to set to a ');
     COMMIT;
END;

Now we have 2 jobs created, one to update THIS_TABLE and one to update my first "rules" table. -10 for using DML in a job. -10 for not putting them into a single job that could fail together. -30 for not creating a stored procedure to do this.

But wait, it's not over yet. We're getting to the very best part I think.

BEGIN
  dbms_job.submit
    ( job => ln_jobno, 
      what => 'insert into errors (name,error_date, text) 
                       values (''wow'', sysdate, ''error turning it back on') ;', 
      next => sysdate+(.5/24));

EXCEPTION
  WHEN others THEN
      insert into errors (name,error_date, text) 
      values ('doing stuff', sysdate, ' MY_ID '||l_my_id||' creating job to log errors');

    COMMIT;
END;

Did I catch a "niner" in there?

Did you catch what that final job did? It "logged" an error from the previous 2 jobs. Really? Do jobs work like that? I'm not really sure that they do.

This little snippet is just a snapshot into my life over the last few years. This kind of thing is everywhere. (I know everywhere!). I've mentioned before, but I've been reading The Daily WTF since 2005. Daily. I've learned more from that site than perhaps any other because you learn what not to do...which is just as important as what to do. It takes years to gain the necessary experience (read: screwing up) to know what not to do, The Daily WTF speeds that up significantly by allowing you to witness others mistakes. We've all made them, to be sure. It's whether we learn from them that is important.

Everything is a Bit Bucket

2010-03-10T13:34:00.002-05:00

By: Michael O'Neill
@oraclenude
oraclenude.crisatunity.com

In response to Chet's frustration over yet another encounter with a database agnostic, I wanted to contribute my first article to the oraclenerd franchise. My thoughts seemed too long for the comment stream.

I ascribe the kernel of thought behind "the database is a bit bucket" primarily to each and every database vendor that ever existed. Every database vendor, in an effort to persuade users of competitive products to adopt their product, has participated willingly in espousing some core aspect of how "same as the other guy" their product is in addition to whatever differentiation pitch they have.

Now, the generally weak-minded and lazy developer (yes, I think the majority of developers are in fact weak-minded and lazy) latches on to the vendor's selective "sameness" claims for professional and personal reasons. (full disclosure: I am both an Oracle DBA and .NET developer)

Professionally, because they are financially invested in writing third-party code not database code. To them, the less they spend learning and understanding the particulars of things like databases, operating systems, networks, human beings, etc. the better. Personally, because there is a dominate thread in the culture of developers to dismiss the database as interesting or meaningful. It is a form of heresy to show affection towards any platform in any specificity.

This is why Java's Big Lie of "write once, run any where" swoons so many. Java's Big Lie is analogous to "the database is a bit bucket" by declaring that even the language of software code should be as absolutely interchangeable as possible - even at the expense of being cost-effective or useful. There is an unquestioned faith that decoupling everything from everything is a good thing. This faith gives us code that is as far from the simplest thing that could work from the first moment writing the code is undertaken. It is a faith I reject. That's why I'm an ORACLENERD.

P.S. I know oraclenude and oraclenerd is confusing. It's supposed to be.

APEX: Create and Parse Arrays

2010-03-10T01:23:00.003-05:00

It's been awhile since I've been able to work with APEX extensively, so I am rusty.

A question came up today whether we could get multiple values into a single variable (Item in APEX).

Yes we can!

APEX_UTILSNeed some data first:

CREATE TABLE t ( some_text VARCHAR2(10) );

INSERT INTO t ( some_text )
SELECT dbms_random.string( 'a', 10 ) some_text
FROM dual
  CONNECT BY LEVEL <= 5;

CJUSTICE@TESTING>SELECT * FROM t;

SOME_TEXT
----------
thrFXviVWJ
kpfGRRwctv
EVxNrcmBHC
gcBlHaKrLa
irYduOZfkS

I want that table data to be in a single item. TABLE_TO_STRING is your function.

VAR C VARCHAR2(100);

DECLARE
  l_table APEX_APPLICATION_GLOBAL.VC_ARR2;
BEGIN
  SELECT some_text
  BULK COLLECT INTO l_table
  FROM t;

  :c := apex_util.table_to_string( p_table => l_table );
END;
/

PL/SQL procedure successfully completed.


C
-----------------------------------------------------------
thrFXviVWJ:kpfGRRwctv:EVxNrcmBHC:gcBlHaKrLa:irYduOZfkS

Easy enough. How about converting it back to a table? STRING_TO_TABLE is your answer.

DECLARE
  l_table APEX_APPLICATION_GLOBAL.VC_ARR2;
BEGIN
  l_table := apex_util.string_to_table( p_string => :c );
  
  FOR i IN 1..l_table.COUNT LOOP
    d( 'value ' || i || ':  ' || l_table(i) );
  END LOOP;
END;
/

value 1:  thrFXviVWJ
value 2:  kpfGRRwctv
value 3:  EVxNrcmBHC
value 4:  gcBlHaKrLa
value 5:  irYduOZfkS

PL/SQL procedure successfully completed.

Done.

The "Database is a Bucket" Mentality

2010-03-09T22:50:00.004-05:00

Front and center again...I just woke up from a nap, I'm grumpy, so I must write. Besides, I haven't had a good rant in quite some time.

Friend of mine asked me last week for some advice, specifically asking if there was a tool to convert Oracle SQL Syntax to the ANSI SQL syntax. (A quick search turned up this (it was the first result), if you're interested).

I had to ask why.

Client is switching to an open source database, i.e. "free." Oracle licensing is way too pricey.

I'm sure Oracle costs a lot of money, it's pretty darn good software. Quite possibly the best in the world especially in the database realm. I've written about the incredibly feature rich goodness that is the Oracle database here here...actually, just trust me. It's in my name.

Why is there even a comparison?

Could it be that everyone out there believes that the sole purpose of a database is to store data? That it can't do anything else? The storage and retrieval of data...that's all it does of course.

It's like saying the Democrats and Republicans are the same...at face value, perhaps, but the devil is in the details.

This, this "Bit Bucket" mentality is what is so incredibly frustrating.

I am no position to argue the differences between the various flavors of database, I lack the experience. But if I were using SQL Server, I would leverage the shit out of it's capabilities. If I were using MySQL, I would leverage the shit out of it's capabilities. If I were using Firebird, I would leverage the shit out of it's capabilities. Same goes for every single flavor out there. Get my point here?

The database is NOT a bit bucket!

Do I need to use more 4-letter words?

I know that Oracle is feature rich and that 99% percent of your code can live in the database...think APEX and PL/SQL. You could probably put ALL of your code inside the database if you wanted to put the javascript in BLOBs as well.

Please, please please quit telling me they are the same...they are not.

Follow up rant by Mr. O'Neill can be found on this following post Everything is a Bit Bucket

Code Comment WTF? Part 209

2010-03-09T01:38:00.002-05:00

Found this in a snippet today:

-- ********************************
-- End of Package Body

END package_pkg ;
/

Seriously? Was that necessary? Could I possibly be under the illusion that it is not the end of the package?

Stop it.

Now.

APEX: LDAP Authentication

2010-03-09T01:09:00.004-05:00

I got called into a discussion about an existing APEX application. The custom LDAP functionality wasn't working as they expected.

I knew APEX had an LDAP authentication scheme (and don't know the full history of the project so I can't (won't) comment on why it wasn't used). So I fired up my local sandbox just to see how easy or hard it was. Admittedly, I have always avoided anything to do with LDAP...not sure why (plate is full?). I used this as a guide.

Anyway, it was remarkably easy.

Setup
APEX: 3.2.1
Web Server: Apache (OHS)
Database:

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Prod
PL/SQL Release 10.2.0.3.0 - Production
CORE    10.2.0.3.0      Production
TNS for 32-bit Windows: Version 10.2.0.3.0 - Production
NLSRTL Version 10.2.0.3.0 - Production

First I fired up the web server:

C:\oracle\http\opmn\bin>opmnctl start
opmnctl: opmn started

C:\oracle\http\opmn\bin>opmnctl startproc process-type=HTTP_Server
opmnctl: starting opmn managed processes...

Opened up APEX, and created a new application. For authentication schemes I chose "No Authentication."

After I had created the application, I went into Shared Components --> Authentication Schemes --> Create

Select the default and click Next

Select "Show Login Page and Use LDAP Directory Credentials" and click Next

I've already done this so I'm selecting my current Login page, 11, click Next

Enter your LDAP Host and your DN:

Your DN String should look something like this (from article above):

cn=%LDAP_USER%,l=amer,dc=oracle,dc=com

Make sure you use the %LDAP_USER% after the cn= portion of the string.

Name it ldap_test, click Create Scheme:

You will then be redirected back to the list of Authentication Schemes, ldap_test should now be current

To test it just run your application and login using your LDAP (AD) credentials

Success!

February Top 10

2010-03-08T13:35:00.005-05:00

As the name implies, it's the Top 10 for this past month. Probably the biggest reason I am doing this is the EBS Install series has become a runaway success. I've never had something become so popular so quickly...which of course tells me I'm no good and John Piwowar is the best ever. :)

John deserves quite a bit of recognition so the more I can provide, the better.

Title	Pageviews	Unique Pageviews
EBS Install Guide - Part I	541	394
Learn Oracle Business Intelligency (OBIEE)	444	337
EBS Install Guide - Part 2	347	262
DBMS_CRYPTO: Example	339	295
APEX: Install 3.2.1 on 11gR2	263	225
EBS Install Guide - Part 3	253	202
OBIEE Posts	243	215
OBIEE: How to Migrate Your rpd	234	183
BULK COLLECT and FORALL	187	166

SQL Developer: Install Unit Testing Repository

2010-03-08T08:41:00.003-05:00

Get the latest SQL Developer release here.

I'm not a big tools guy, I prefer SQL*Plus to anything else. I especially don't like paying for tools (yes, the database is a tool and costs a lot of money...I do realize the hypocrisy).

After Syme Kutz's presentation at SOUG, I've been looking more closely at SQL Developer. I've been using it (and JDeveloper) since they were both made freely available a few years ago. Mostly for the schema browsing, looking around, importing and exporting data. I do use it (SQL Developer) to write reports that I can share with the Business folks as well.

Syme's presentation was primarily on Unit Testing (which I begged for). First step to using Unit Testing is to install the repository, a set of tables the application uses to build and store tests and their results.

You need to have version 2.1 or greater.

First up, go to Tools --> Unit Test --> Select Current Repository

You'll be prompted to select a connection (i.e. database) to use

Would you like to create one now? Select Yes.

You're then told the the required roles do not exist, select OK.

Confirm running SQL

Running...will take just a few seconds

Success!

That's it. Easy right? Future posts will detail managing users and creating tests.

OBIEE: Default Answers Template?

2010-03-07T16:31:00.003-05:00

After trying out the lazyweb method of search (aka Twitter) and not getting much help, I resorted to help at the OTN OBIEE Forum. It's not Twitter's fault, I think this problem was a bit too complex to describe in 140 characters.

Here's the post on OTN. I started to get nervous too, I posted on Friday and hadn't had a response...until today. 3 days? Man...that's way too long!

Here's the short of it.

Our reports were coming out funny. Dimension column headings had one style and the Fact table column headings had another.

Using Firebug, I could easily isolate the sections.

On the Dimension column, the definition looked like this:

<th  
  class="ColumnHdg" 
  style="background-color: rgb(231, 231, 247); font-size: 9px; color: rgb(0, 51, 102);" 
  scope="col" 
  dir="ltr">Product Desc
</th>

The Fact table column was defined as:

<th 
  class="ColumnHdg" 
  scope="col" 
  dir="ltr">Basis Amount
</th>

Note the style attribute...that overrides any class settings. Very annoying.

I thought it would be relatively simple to fix. I worked with custom messages before, this had to be similar. So I began to "grep" the messages directory

c:\oraclebi\web\msgdb\>findstr /i /m /s /c:"background-color" *.*

Nothing.

How about looking for the name of the class, ColumnHdg?

c:\oraclebi\web\msgdb\>findstr /i /m /s /c:"columnhdg" *.*
messages\criteriatemplates.xml
messages\formattemplates.xml
messages\mktgadminuitemplates.xml
messages\mktgcommontemplates.xml
messages\mktglistformattemplates.xml
messages\mktgsegmenttemplates.xml

So I start with criteriatemplates.xml and find the reference to columnHdg (just now realizing that the case doesn't match...oh well). That was part of the WebMessage kuiColumnFormulaEditor. So I searched for that...

You see where I'm going.

That lead me to the javascript files (of which there are tons). Nothing...not a single thing that could possibly be adding this style attribute.

That's when I mapped the dev server drive to my local computer and opened up WinMerge

I then began to compare every single file in both the msgdb and res (javascript) folders. I would then compare the files that were different to see if that could be the cause. Still...nothing.

I had looked in the webcat before, but couldn't find anything of global significance. I was headed back there though since I had lost hope with custom messages and/or javascript.

Then I got a message from the OTN Forum Administrator...could it be? Looked at the name of the person who answered it first, John Minkjan...sweet!

Looks like you forgot to reset the OOB settings when installing OBIEE:
have a look here to reset them:
http://obiee101.blogspot.com/2009/02/obiee-editing-system-wide-defaults.html

Click through, follow his instructions, bounce the server and voila!

As of this writing, I don't know what OOB stands for; I could make something up, but it probably wouldn't be appropriate for this family site.

Thanks John!

OBIEE: XML File as Data Source

2010-03-03T15:48:00.000-05:00

I received a question in the form of a comment on using XML files here. Fittingly, we were discussing this exact topic yesterday and I decided I would try it out.

So here goes.

You'll need 2 files: a XML file and a XSL stylesheet (which defines the XML file). I found said files here. I'll also provide them here (cdcatalog.xml) and here (cdcatalog.xls).

In the Administration tool, go to File --> Import --> from Database

When you prompted to select a datasource, select XML and the window should look like this

For URL, browse to the location you saved cdcatalog.xml, do the same for XSLT

Hit OK and you'll be prompted with the Import dialog

Import the "table" and you're done.

Learning About Performance

2010-03-03T00:37:00.004-05:00

I've found myself at a certain disadvantage lately, specifically in regards to understanding tuning. I don't like being at a disadvantage...I'm competitive that way.

I understand, from a very high level of what needs to be done, but I could not prove my theories. Well, I could, sort of, but it wasn't necessarily a reasoned or logical approach. I just tried all possible permutations. There's a certain time constraint with that method and when you're talking about huge volumes of data (i.e. datawarehouses), time is short.

I've decided it's time to change that, so I'll begin to peck away ever so slowly.

In Usage Tracking, there is a table called S_ETL_TIME_DAY. It's a sister table to S_ETL_DAY which is your everyday time dimension. S_ETL_TIME_DAY breaks down a single day into hours and minutes which means there are 1440 records (24*60*60).

Somewhere I saw the following SQL:

SELECT DISTINCT hours
FROM s_etl_time_day

which just returns the hours in the day (24). I've pondered on whether DISTINCT is a bug, but it seems fairly innocuous here.

What about this though?

SELECT rownum - 1 hours
FROM dual
  CONNECT BY LEVEL <= 24;

Which one is faster?

Let's try an explain plan

Query 1

S_NQ_SCHED@TESTING>EXPLAIN PLAN FOR
  2  SELECT rownum - 1 hours
  3  FROM dual
  4    CONNECT BY LEVEL <= 24;

Explained.

Elapsed: 00:00:00.05
S_NQ_SCHED@TESTING>@EXPLAIN

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------
Plan hash value: 1731520519

------------------------------------------------------------------------------
| Id  | Operation                     | Name | Rows  | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |      |     1 |     2   (0)| 00:00:01 |
|   1 |  COUNT                        |      |       |            |          |
|*  2 |   CONNECT BY WITHOUT FILTERING|      |       |            |          |
|   3 |    FAST DUAL                  |      |     1 |     2   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter(LEVEL<=24)

15 rows selected.

Query 2

S_NQ_SCHED@TESTING>EXPLAIN PLAN FOR
  2  SELECT DISTINCT hours
  3  FROM s_etl_time_day;

Explained.

Elapsed: 00:00:00.04
S_NQ_SCHED@TESTING>@explain

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------
Plan hash value: 878743397

-------------------------------------------------------------------------------------
| Id  | Operation          | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |                |    24 |    72 |     5  (20)| 00:00:01 |
|   1 |  HASH UNIQUE       |                |    24 |    72 |     5  (20)| 00:00:01 |
|   2 |   TABLE ACCESS FULL| S_ETL_TIME_DAY |  1440 |  4320 |     4   (0)| 00:00:01 |
-------------------------------------------------------------------------------------

9 rows selected.

Sadly, about the only thing I can usually see in an explain plan is whether an index is being picked up or if there is a nested loop. I'm not going to worry about all of it now...this is just a start. More to get me in the habit.

Next I set up tracing since "reading" the file is so much easier now.

ALTER SESSION SET TRACEFILE_IDENTIFIER = 'HOURS';

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 12';

Here's the output:

Query 1	Query 2

What does it all mean? I am sure you know or understand better than I. For me, I just need to create the habit. I'll read (more) about the specifics of the explain plan and the output from tkprof. If you want to explain, feel free. If you want to do it in your own post, link it up or you can write it here. I'd be happy to host it.

COLLABORATE 10

2010-03-02T22:25:00.000-05:00

Today I got notice that I would be granted a blogger pass for COLLABORATE 10 in Las Vegas, NV. Vegas baby, Vegas! What's more, all three groups, IOUG, OAUG and Quest approved.

If you weren't aware, I had the same deal last year through OAUG. I realized, very quickly, that living up to the obligation was very tough. I wrote 13 articles over that short span of time and it was completely exhausting.

So of course I want to do it again...I'm a glutton for punishment I guess.

DBA_TABLES vs DBA_OBJECTS

2010-03-02T10:38:00.002-05:00

CJUSTICE@TESTING>SELECT * FROM V$VERSION;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Prod
PL/SQL Release 10.2.0.3.0 - Production
CORE    10.2.0.3.0      Production
TNS for 32-bit Windows: Version 10.2.0.3.0 - Production
NLSRTL Version 10.2.0.3.0 - Production

Last night I was installing the Unit Testing repository for SQL Developer for a fun little post. After installing the repository, I just did a brief sanity check.

CJUSTICE@TESTING>SELECT owner, COUNT(*) c
  2  FROM dba_objects
  3  GROUP BY owner
  4  ORDER BY 1;
  
  OWNER                                   C
------------------------------ ----------
...snip
SI_INFORMTN_SCHEMA                      8
SYS                                 22970
SYSMAN                               1341
SYSTEM                                454
S_NQ_SCHED                              3
TSMSYS                                  3
WMSYS                                 242
XDB                                   682

27 rows selected.

Strange.

I created a user, UNIT_TESTING, to house the data and fed it to SQL Developer. Did I create the user?

CJUSTICE@TESTING>SELECT COUNT(*) 
FROM dba_users 
WHERE username = 'UNIT_TESTING';

  COUNT(*)
----------
         1

Yeah.

I check DBA_OBJECTS using UNIT_TESTING as the predicate:

CJUSTICE@TESTING>SELECT * FROM dba_objects WHERE owner = 'UNIT_TESTING';

no rows selected

Really?

CJUSTICE@TESTING>SELECT table_name
  2  FROM dba_tables
  3  WHERE owner = 'UNIT_TESTING';

TABLE_NAME
------------------------------
UT_LIB_TEARDOWNS
UT_LOOKUP_CATEGORIES
UT_LOOKUP_DATATYPES
UT_LOOKUP_VALUES
UT_METADATA
UT_TEST
UT_TEST_ARGUMENTS
UT_TEST_IMPL
UT_VALIDATIONS
UT_TEST_IMPL_ARGUMENTS
UT_LIB_STARTUPS
UT_LIB_VALIDATIONS
UT_LIB_DYN_QUERIES
UT_SUITE
UT_SUITE_TEST
UT_TEST_IMPL_VAL_RESULTS
UT_TEST_IMPL_ARG_RESULTS
UT_TEST_IMPL_RESULTS
UT_TEST_COVERAGE_STATS
UT_TEST_RESULTS
UT_SUITE_RESULTS
UT_SUITE_TEST_RESULTS

22 rows selected.

WTF?

Does this mean that my data dictionary is corrupted? This is a sandbox so it is very well possible...just never seen this kind of thing before.

OBIEE: Dynamic Variables

2010-03-01T22:30:00.000-05:00

Apparently this is a 3 part series, Part I, OBIEE: Text File as Data Source and Part II, OBIEE: Create Repository Init Block.

Finally, to the ultimate goal, dynamic session variables. Ideally, these would be repository variables, but since I'm using the row-wise variety, that is not possible (as of 10.1.3.4). Once a day, or some other relatively long time frame would be sufficient.

Anyway, there's an occasion where dynamically creating variables might be needed or necessary.

To recap the problem

Due to certain restrictions in our environment, the tnsnames entries in our separate environments (Dev/QA/Stage/Prod) are exactly the same. I would prefer to have one entry in each environment that points to that environments database, but alas, I cannot. What this does is forces us to change the password in the connection pools when migrating. I know it's not ideal, but it is what it is.

The proof of concept I am embarking on will allow us to lookup the tnsname entry and the associated password for that environment, theoretically, requiring no changes between environments.

In that respect, it was decided that we would have a single service account. From that service account, we could look up the values we need for each environment. Here's the other fun problem...what comes first? The chicken or the egg? This service account password will be stored in a simple text file that will be locked down by the system administrators. Don't ask what happens when the password changes either. At this point, it's just a proof of concept. So be gentle. If you have an established way to manage passwords...please share.

A few people did chime in, Pete Scott suggesting using the a command line utility detailed here. Christian chimed in that I should use it at my own risk and rnm1978 said this problem has not been truly solved yet.

The whole issue of migration OBIEE deployments through the environments is a bit of a pig's ear with no foolproof method that I've seen

I need to look up that phrase as I'm certain it's peculiar to my brethren across the pond.

Finally, the meat of the post. Here's my table:

OBI@TESTING>@desc obi_variables
 Name                           Null?    Type
 ------------------------------ -------- -------------
 VARIABLE_NAME                  NOT NULL VARCHAR2(50)
 VARIABLE_VALUE                 NOT NULL VARCHAR2(250)
 VARIABLE_TYPE_CODE             NOT NULL VARCHAR2(30)

I'll throw a few records in there for fun (and to obviously demonstrate).

INSERT INTO obi_variables
  ( variable_name,
    variable_value,
    variable_type_code )
VALUES
  ( 'TNS_TESTING',
    'TESTING',
    'TNSNAME' );

INSERT INTO obi_variables
  ( variable_name,
    variable_value,
    variable_type_code )
VALUES
  ( 'PW_TESTING',
    'TESTING',
    'PASSWORD' );

INSERT INTO obi_variables
  ( variable_name,
    variable_value,
    variable_type_code )
VALUES
  ( 'URL_TESTING',
    'http://localhost:7777/pls/apex/f?p=101:1',
    'URL' );

TYPE       VARIABLE_NAM VARIABLE_VALUE
---------- ------------ ------------------------------------------
PASSWORD   PW_TESTING   TESTING
URL        URL_TESTING  http://localhost:7777/pls/apex/f?p=101:1
TNSNAME    TNS_TESTING  TESTING

For demonstration purposes, I am going to create a new database in the Physical layer and add a connection pool called obi variables pool. From the previous post, I'll be using the values that I brought in through the text file. The only difference is that I added username...just in case. To sum that up, I'm bringing in the TNSNAME entry, the USERNAME and the PASSWORD.

Your connection pool should look like this:

The password column is the same format as the others, VALUEOF( INIT_PASSWORD )

You'll have to re-enter that when you save.

Now I'll be creating an init block using this connection pool. I won't drag you through that entire process again (your welcome). This is a Session Init Block as opposed to the Repository Init Block in the previous example.

The only difference is that I will be using a row-wise variable. I'll get to that in a second.

After selecting the obi variable pool connection pool, entering the following SQL statement into the area provided:

SELECT variable_name, variable_value
FROM obi_variables

.In the Edit Data Target area, select the row-wise initialization variable. Select OK.

Test it and voila! You now have dynamic variables in OBIEE.

OBIEE: Single/Detail Record View

2010-02-28T23:55:00.002-05:00

Recently, I've been searching for a Single Row/Detail type report in Answers. Sadly, there isn't one so named.

Here's what I could find in Answers:

Nothing jumps out at me, so, Twitter to the rescue again

Within minutes...I had an answer from Mr. Christian Berg. What was that answer? The Narrative View.

Here's the screen for the Narrative View:

According to Mr. Berg, you reference the columns using the @1 (@2, @3, etc) syntax, the number representing the column number. So I tried that

Run it...

That's not what I want...

OK, so let's see if I can add some HTML to it. First I check the Contains HTML Markup box and then I enter in the following in the Narrative Text Area:

@1<br>

@2<br>

@3<br>

@4<br>

@5<br>

@6<br>

@7<br>

@8<br>

@9<br>

Here's what it looks like:

Better, but not exactly what I want. Now that I know I can use HTML though, it should be very easy.

Voila!

Random Things: Volume#16

2010-02-28T22:06:00.004-05:00

EBS Challenge
John Piwowar wrote the first part of the EBS Installation Guide back in December, since then, it's become a runaway success. I hadn't realized how so until recently when I was looking at Google Analytics and it came it at number 19, for all time. Now it's up to 13 and with the addition of it to the front page, it should only go up faster.

I've never had a post that was this popular so I thought it fitting that John should get the recognition that he deserves. If you decide to take part and then write it up, let me know and I'll link you up to The EBS Challengepage. I think there are 4 people that have completed it so far. I'll even give you space here if you don't have your own blog.

Travel
Got the best color car ever this week.

OBI EE
For some reason my original, Learning Oracle Business Intelligence (OBIEE) post, has been near the top of the charts for multiple search phrases.

So I did what any sane person would do, I updated it with some new information. Not really sure if it's new, but it's got more. If people are finding it, I might as well update it as often as possible.

SOUG: SQL Developer with Syme Kutz

2010-02-26T00:47:00.003-05:00

Tonight was the Suncoast Oracle User Group (SOUG) meeting with Syme (pronounced Sim-e, I thought it was Si-me) Kutz of Oracle presenting on SQL Developer, mainly the new Unit Testing functionality.

Unfortunately, I missed the first half of the meeting due to a flight delay, but from what I did see, it's very cool. If you read the announcement last week, you'll remember that Kris Rice had offered up (aka - threw under the bus) Syme. I made first contact and then passed the baton to our meeting coordinator who finalized the arrangement.

If you want to check out the Unit Testing features, you need the latest release (2.1), which can be found here. To access it, go to Tools --> Unit Testing

I won't go into gory details simply because I need to use the dang thing first. I'm sure I'll have some posts in the near future.

Anyway, what I did see was pretty slick.

Syme then gave us some history of the product (developed originally by himself and Mr. Rice) and explained a bit more about some of the functionality. Many of you already know about the integration with APEX (I don't know much, other than it exists). That's about to be expanded and will give even more control over many aspects of APEX, including some pretty tight integration with the Unit Testing module.

One really cool thing that he mentioned, if you open up a trace file in SQL Developer, you get a pretty report for it. Apparently reverse engineered from tkprof.

First, find your trace file:

Double click it to open it and you'll see something like this (you'll have to click through on this one):

I will break it down if you're too lazy though.

The first column of the report is the SQL:

Next up are the statistics:

Waits:

and finally Row Sources:

Pretty slick stuff.

Thanks Syme for coming down, hopefully we can get you down here again to show us the rest.

OBIEE: Create Repository Init Block

2010-02-24T22:17:00.004-05:00

My post earlier today, OBIEE: Text File as Data Source , was the first part of my overall goal.

Here's what I'm trying to do. Due to certain restrictions in our environment, the tnsnames entries in our separate environments (Dev/QA/Stage/Prod) are exactly the same. I would prefer to have one entry in each environment that points to that environments database, but alas, I cannot. What this does is forces us to change the password in the connection pools when migrating. I know it's not ideal, but it is what it is.

The proof of concept I am embarking on will allow us to lookup the tnsname entry and the associated password for that environment, theoretically, requiring no changes between environments.

In that respect, it was decided that we would have a single service account. From that service account, we could look up the values we need for each environment. Here's the other fun problem...what comes first? The chicken or the egg? This service account password will be stored in a simple text file that will be locked down by the system administrators. Don't ask what happens when the password changes either. At this point, it's just a proof of concept. So be gentle. If you have an established way to manage passwords...please share.

We've pulled the text file into the RPD already. Now, I need to create 2 repository variables to hold these values. I open up the Administration tool, go to Manage then variables. You should see this:

Then follow this picture to create a new Initialization Block

Name your Initialization block init_block_test

Leave the scheduler stuff alone for now, click on Edit Data Source

First, browse for the connection pool

Select the get_local_password (yes, the name mysteriously changed from the default Connection Pool to "get_local_password," it makes sense doesn't it?).

In the Default Initialization String text box, enter:

SELECT username, password 
FROM test.txt

You can test it if you want, but I'm saving it for the final step.

Your Initialization Block Data Source should look like this

Now select Edit Data Target

Which looks like this

Select New and enter INIT_USERNAME in the Name text box and make the Default Initializer 'USERNAME' (in single quotes).

Do the same thing for the password column, INIT_PASSWORD and 'PASSWORD'

Now you are back at the main screen and the Test button (lower left hand corner) should be enabled. Select it.

One final test...let's see if we can access these in the presentation layer.

So it's a success right?

After I was done, I began to think..."You can access the repository variable from the presentation server"

Ummm...that's not good. Double checked the properties of both the Initialization Block and the Variables to see if there's a way to lock it down...and there's not. Because of this "small" little security issue, I'm not completely sold. It is a proof of concept, perhaps it should stay there.

OBIEE: Text File as Data Source

2010-02-24T17:30:00.000-05:00

I have a requirement to pull data from a text file. OBI has the ability to pull from (just about) any database, but can it pull from a text file? Of course.

First I'll create a simple file, call it test.txt. In the file I put

cjustice testing

that "space" between cjustice and testing is actually a tab character (usually represented as -> if you have view formatting turned on). Knowing that it is a tab will be important shortly. Save the file to your root folder, i.e. c:\test.txt

Next up, go to your ODBC Datasources:

Control Panel --> Administrative Tools --> Data Sources (ODBC)

When that opens up, navigate to the System DSN tab and click on Add.

You'll be prompted to choose a driver, select the Microsoft Text Driver (*.txt, *.csv)

Click on Finish.

Next up, you should see the ODBC Text Setup screen, click on the Options tab to expand the window.

It should look like this:

Uncheck the Use Current Directory checkbox (default is usually c:\windows\system32):

Then select the Select Directory button and navigate to your root folder, or the place where you saved the test.txt file. You should see your file, greyed out, in the left hand pane.

Now, at the bottom of the window, uncheck Default (*.*) which should enable the Define Format button

Select the Define Format. For the Format drop down, choose Tab Delimited. Rows to Scan, in my case, will only be 1. You can leave it at the default if you want though.

After you have done that, the Guess button should be enabled for you, go ahead and click it.

See F1 and F2? Those are your columns. Let's name them something meaningful so we can easily reference them later.

In the Name box, it says F1, enter in USERNAME and click on Modify (the button to the right):

Do the same thing for the F2 column, name it PASSWORD.

It should end up looking like this:

Select OK and then select OK again in the ODBC Text Setup screen. You should now see your new data source in the System DSN window.

OBI EE Admin Tool
Now, we need to pull this new data source into the RPD. Open up your RPD, go to File --> Import --> from Database

You'll see this window, go ahead and select your new data source:

You'll be prompted with the Import window, expand the c:\ drive and then find your file

Select Import (at the bottom), wait for it to complete and then close the window. In your physical layer you should see a new "database" with the name "text_test" (or whatever you named your data source). Go ahead and expand that to see what you've got:

To verify, right click on the table "test.txt" and select View Data.

All done.

Connect to HP Neoview using Heterogenous Services

2010-02-23T22:49:00.002-05:00

Being the lazy sort that I am and not wanting to pay for a SQL client, I decided to use my local Oracle instance to access a HP Neoview database. Please don't ask me any questions about it, because like much of the world, I don't know either.

Hat(s) off to Tak Tang. I've used his guide a number of times throughout the years. If he blogs, I can't find it, so if you know about it, please link him up. The first time I used it was back in aught (sp?) 5 to connect to a DB2 instance.

Setup
Database (source): Oracle 10gR2
OS: Windows Vista Ultimate running as a VirtualBox Guest on Ubuntu
Database (target): HP Neoview 2.4 (or something)
OS: Doesn't matter

Materials
HP Neoview driver which can be found here. (Thanks Christian)

Steps
Download the HP Neoview driver and install it. Since it is windows, just click away accepting all the defaults.

Next, configure an ODBC Datasource. I won't go into the gory details, but it's pretty easy. When you are done, you should see something like this:

Now comes the fun part.

In your <ORACLE_HOME>\hs\admin directory you should see the following:

Open up the inithsodbc.ora file and set the parameters as follows:

HS_FDS_CONNECT_INFO = NEOVIEW
HS_FDS_TRACE_LEVEL = 1

Save the file as initNEOVIEW.ora in the same directory.

Now traverse to <ORACLE_HOME>\network\admin and open up your listener.ora file. Add a SID_DESC so that your file now looks like this:

LISTENER=
  (DESCRIPTION=
    (ADDRESS_LIST=
      (ADDRESS=(PROTOCOL=tcp)(HOST=localhost)(PORT=1521))
      (ADDRESS=(PROTOCOL=ipc)(KEY=extproc))))
SID_LIST_LISTENER=
  (SID_LIST=
    (SID_DESC=
      (GLOBAL_DBNAME=testing)
      (ORACLE_HOME=c:\oracle)
      (SID_NAME=testing))
    (SID_DESC=
      (SID_NAME=plsextproc)
      (ORACLE_HOME=c:\oracle)
      (PROGRAM=extproc))
    (SID_DESC=
      (SID_NAME=NEOVIEW)
      (ORACLE_HOME=c:\oracle)
      (PROGRAM=hsodbc)))

Reload or stop and start your listener.

Now open up your tnsnames.ora file and add an entry for NEOVIEW. It should look like this:

NEOVIEW =
  (DESCRIPTION=
    (ADDRESS_LIST=
      (ADDRESS=
        (PROTOCOL=TCP)
        (HOST=localhost)
        (PORT=1521)
      )
    )
    (CONNECT_DATA=(SID=NEOVIEW))
    (HS=OK)
  )

The last step is to create a database link.

CREATE DATABASE LINK neoview
  CONNECT TO "username"
  IDENTIFIED BY "password"
  USING 'NEOVIEW';

You should be all set now. You can test it out by issueing a simple

SELECT * FROM dual

If that works, you're done.

If you have problems, check out Tak's Troubleshooting section.

OBIEE: Call SAPurgeCacheByDatabase

2010-02-22T22:08:00.003-05:00

So the reason I am investigating the ODBC Function extensions is that we are trying to tie this into our existing ETL process. After load, and within a designated time frame, we'd like to clear the cache for a given set of tables. Fortunately, all those tables exist in a single database. Unfortunately, I'm wasting your time by putting these into separate posts. :)

First, nothing up my sleeve.

So you understand the mapping, I made a pretty picture for that (it's not readily available in the cache manager)

I go off and run the reports (no need for screenshots of those).

Refresh the cache manager

So now I have cache entries for both "databases" (in quotes because they are actually the same physical database, just 2 separate connections).

I then create the SQL file and put the following:

CALL SAPurgeCacheByDatabase( 'local-sh' );

Then I just follow the same procedure as SAPurgeCacheAll()

nqcmd -d AnalyticsWeb -u Administrator -p Administrator -s sapurgecachebydatabase.sql

The output is almost identical as well

-------------------------------------------------------------------------------
          Oracle BI Server
          Copyright (c) 1997-2009 Oracle Corporation, All rights reserved
-------------------------------------------------------------------------------

CALL SAPurgeCacheByDatabase( 'local-sh' )
CALL SAPurgeCacheByDatabase( 'local-sh' )
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
------------------------------
RESULT_CODE  RESULT_MESSAGE

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
------------------------------
1            [59118] Operation SAPurgeCacheByDatabase succeeded!

-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
------------------------------
Row count: 1
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
------------------------------

Processed: 1 queries

And voila! The Sales cache has been cleared.

OBIEE: Call SAPurgeAllCache()

2010-02-22T11:58:00.003-05:00

Just a note to myself on how to clear the entire BI Server cache.

Mr. Minkjan has a great article here and Gerard Nico has one here.

Caching has been enabled (via <BI_SERVER_HOME>\server\Config\NQSConfig.ini)

For a list of ODBC Function extensions, read here.

First up, I run a simple report and then verify that there is an entry in the cache

I then create a small SQL script and put it in my c:\ folder:

(OK, just ignore what it says in the Title bar, I moved it)

Open up a cmd prompt and issue the following:

nqcmd -d AnalyticsWeb -u Administrator -p Administrator -s purgeallcache.sql

Hit enter and you should see the following:

-------------------------------------------------------------------------------
          Oracle BI Server
          Copyright (c) 1997-2009 Oracle Corporation, All rights reserved
-------------------------------------------------------------------------------

CALL SAPurgeAllCache()

CALL SAPurgeAllCache()

------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
------------------------------
RESULT_CODE  RESULT_MESSAGE

------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
------------------------------
1            [59118] Operation SAPurgeAllCache succeeded!

------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
------------------------------
Row count: 1
------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------
------------------------------

Processed: 1 queries

Check the Cache Manager and voila!

For future reference, here are the options available for the nqcmd utility:

Command: nqcmd - a command line client which can issue SQL statements
                 against either Oracle BI server or a variety
                 of ODBC compliant backend databases.
SYNOPSIS
         nqcmd [OPTION]...
DESCRIPTION
         -d
         -u
         -p
         -s
         -o
         -D
         -C<# number of fetched rows by column-wise binding>
         -R<# number of fetched rows by row-wise binding>
         -a (a flag to enable async processing)
         -f (a flag to enable to flush output file for each write)
         -H (a flag to enable to open/close a request handle for each query)
         -z (a flag to enable UTF8 instead of ACP)
         -utf16 (a flag to enable UTF16 instead of ACP)
         -q (a flag to turn off row output)
         -NoFetch (a flag to disable data fetch with query execution)
         -NotForwardCursor (a flag to disable forwardonly cursor)
         -SessionVar =

SOUG: SQL Developer Unit Testing

2010-02-21T23:12:00.003-05:00

This week the Suncoast Oracle Users Group will be hosting Syme Kutz, Senior Architect for Database Tools at Oracle.

Update
I received Syme's bio (from Syme of course). It's pretty impressive...

I started working at Oracle in October of 1995 in the Systems Performance Group of Consulting under Cary Millsap.

I then spent 8 years tuning the Database and Oracle Applications. My Experience tuning application lead me to work with Max Schierson fixing and Tuning iStore. I left consulting and moved into Applications It working for Max at Headquarters were our focus was building custom applications to better facilitate oracle policies and programs. I built custom applications until a position opened up on APEX the development team. After rebuilding the Database management side of APEX, Kris Rice and I began the Sql Developer project. When the group split and Sql Developer became a product I followed. I have been a developer on Sql Developer since then building various functionality, such as reports and Unti Testing.

We "found" Syme through Kris Rice who unceremoniously offered his services up on Twitter.

After much wrangling and negotiating, we finally managed to talk Syme into coming down from Orlando.

According to Kris, Syme was heavily involved with the new Unit Testing functionality of SQL Developer. We all know how much testing us database folks do, so it makes perfect sense right?

Anyway, if you're in town and Thursday, please come by and check out Syme's presentation, it should be very interesting.

Database Application Security - A Visualization

2010-02-21T18:17:00.003-05:00

Security, in regards to the database, is pretty broad. There are a multitude of ways to secure your database. From physical or logical access to SQL Injection.

This is a (not so) pretty picture of how I visualize security from an application point of view.

The base of any application are the tables. Access to those tables should be limited to database views and PL/SQL packages (API) that are owned by the application (schema).

SELECT privileges on those views should be given to only users who need it (users also includes other applications). Views present the data in the format that is required by the application. VPD (column or row) can be used to further restrict access to sensitive data.

EXECUTE privileges on the APIs (PL/SQL) should only be given to those applications that need it. Under no circumstances, should direct INSERT/UPDATE/DELETE privileges be given to another user.

Much of my thoughts on this matter are directly traceable to reading AskTom for so many years. I can't point to any one specific article, I think it's the accumulation/internalization of reading for so long.

It makes sense though.

Who better to decide the functions (INSERT/UPDATE/DELETE) that can be performed against a given set of tables than the person(s) who created them? If said person has left, make someone else responsible for them. I've fought hard over the years to implement this "pattern" and have been fairly unsuccessful. It is difficult to go against years of doing it one way (full access to the underlying tables) to forcing someone or something to use the API.

The idea to visualize it came to me recently and I need to put it down.

What do you think? Am I full of it? Is this reasonable? Anything I left out?

OBIEE: Usage Tracking Performance Improvements

2010-02-17T23:10:00.002-05:00

Usage Tracking, in OBIEE, allows you to capture some basic statistics on your Presentation layer.

I mused recently about diving into the "internals" of OBIEE, specifically learning how to read the logs (Execution Plans) and other stuff.

Mr. Mark Rittman was kind enough to share his thoughts on the matter in the comment section. I'm reposting here because I believe it is valuable insight.

Regarding the thoughts on OBIEE performance. Here's my thinking on the subject.

When I found out about concepts such as OBIEE "driving tables", level 5 logging and BI Server execution plans, my first reaction was to try and dissect them, try and understand them in the same way that I understand the various types of Oracle joins (hash/nested loop etc), Oracle trace files, 10035 traces etc. At the end of the day the BI Server is a virtual database engine and it makes sense to understand and tune it in the same way that we do with Oracle.

But then, the reality is, if you are spending time understanding level 5 log files, the various federated query optimization techniques, to be honest you'd be better off trying to optimize the underlying data so that the BI Server didn't then have to use these features.

For example - if you're trying to optimize a federated query, you should try and co-locate the data instead, either by copying both data sets to the same physical database, or better still, combine them into a data mart or warehouse. If you are having problems with aggregation and start thinking about the Aggregate Persistence Wizard - well you'd be better off putting the data into Essbase instead, and run OBIEE against that rather than your detail-level database.

And so on. So whilst it's interesting to see how the internals of the BI Server work, how it joins data, how you can optimize cross-database joins etc, in the end this is more from the intellectual angle, as you get far more "bang for your buck" by sorting out the data layer away from OBIEE and then bring a more optimized set of data into the BI Server. "Less is more" is my perspective with OBIEE, and the more that we can do at the (Oracle) database or (Essbase) OLAP server end, and keep the BI Server model simple, is the better in my book.

Just my 0.02

That last paragraph is key. Push the data into the database and keep the OBIEE model simple. Sounds right up my alley!

With Mr. Rittman's comments swirling in my head, I went about my performance tuning a bit differently.

We added some user tables to the Usage Tracking model, which seem to bring the performance to a screeching halt. A halt is defined at about a minute or so for the complete page to render...well, the page renders, I guess I mean the queries to return.

Here's essentially what it looks like:

Two databases and three separate schemas.

The very first thing I did was limit the number of rows returned to the BI Server from S_NQ_ACCT. Looking at the data more closely, it was pulling in more than just the reports, it had stats for ValuePrompts and Filters as well. Those weren't needed. I created a view on top of S_NQ_ACCT which returned about a quarter of the records. For perspective, I'm dealing with about 400K records.

Added that to the Physical Layer (File --> Import --> From Database) then updated the sources.

Ran the offending page...no difference.

Looking at the logs, I can see 4 queries being sent to the 2 databases. It appears that OBIEE won't push the join to the database if using separate connection pools. After thinking about it, it made perfect sense, OBIEE can't assume that each connection has the same privileges as the other. (Though I cannot confirm that yet, just seems logical).

Next, I pulled the data from Schema 2 into Schema 1.

Ran the page, no difference.

Checked the logs, and there were still 4 queries sent to the 2 databases. To be fair, I may have run the page prior to the changes being propogated to the BI Server.

OK...let's pull the data from DB 2 and see what happens. Import the table, configure everything.

Ran the page, no difference.

Looking at the logs this time, I saw 3 queries. Better right?

For fun, I decided to put the results of Query 2 and Query 3 into a table. Configured everything...blah blah blah.

Ran the page, it can up almost immediately. I called 2 people and had them run the same report and they saw the same behavior. Win! (Caching is not turned on in this environment).

When I looked at the log, a single query was sent to the database. It was an ugly query, but the joins took place in the database as opposed to the BI Server.

This won't be the final implementation, I'll probably have to back out these changes. But it's a great proof of concept.