Tuesday, December 9, 2008

Application Developers vs. Database Developers: Part II

You can read the first article here. My application developer friend, Mr. M, emailed me and another fine gentleman this little blurb recently:

Mr. M:
OH YEAH BABY!!! TEN TIMES FASTER!!!! YEAH!!!!!!!!

Hey seriously, what a tub of shit Oracle is. Where does this myth come from that it's such a great platform? Their client tools suck balls and it's generally just a pain in the ass to work with from a developer's point of view. But devs for some reason are under this impression that from thew server perspective it's rock solid and performant. Well, it may be solid, but it's a fucking turd. Our dba here - definitely an Oracle guy - has been tasked with looking into moving us onto an oss db. He basically refuses to even look at MySQL, stating that it's a mickey mouse worthless pile of shit (can't really argue with him there lol), so that basically leaves Postgres. So it's been a few weeks now, and he will basically admit now that Postgres completely waxes Oracle as far as performance goes. LOL We run RAC in production too. He's looking at failover, replication, blah blah blah now, we'll see what the verdict on that is. Oh, and Oracle AQ? That's a worthless pile of shit too. Why do they tack a fucking message queue that doesn't play nice with jms onto the fucking database, which is probably already overworked? Oh wait, that's right, they're in the business of selling per cpu licenses! Cocksuckers.
This was prompted by a recent Oracle email blast about the Exadata storage system/Warehouse.

As I did before, I'll just put the email here.

Me:
Agreed, their client tools aren't all the great. Which ones are you using?

I use SQL*Plus (naturally), SQL Developer and JDeveloper. The latter 2 tie in very nicely with Subversion. With JDeveloper, when I want to try out Java, it's got a pretty good suite of tools.

Oracle starting out catering to businesses, Microsoft started with the consumer. Oracle has made pretty good strides in making their software more usable while Microsoft has made theirs more scalable.

I haven't used AQ a whole lot and definitely not with Java. I do know that it's billed as JMS compliant.

Postgres has it's place and so does Oracle. It is a great platform if you know how to leverage it. PL/SQL is a 3GL (or is it 4) and you can write entire applications using it...and if you do that right, an incredibly robust and scalable application at that.
Mr. M:

"It is a great platform if you know how to leverage it. PL/SQL is a 3GL (or is it 4) and you can write entire applications using it...and if you do that right, an incredibly robust and scalable application at that."

NO!!! NO!!! NOOOOO!!!

I want to beat people like you who say this with a ball pean hammer. There are only a select few individuals on this earth who can write and read application logic written in SQL. AVOID THIS ANTI-PATTERN AT ALL COSTS! What is it with you f_cking database guys??? Why do you always want to stuff as much crap into the db as possible?

DUDE! SQL IS A PILE OF SHIT FOR EXPRESSING APPLICATION LOGIC!!!

It's fine for querying and manipulating sets of data, in a relational database. But it is a worthless sack of shit for expressing application logic!

I'm having to dig through this f_cking abortion right now because some Oracle f_ckhead thought "you can write entire applications using it!" Blog that, mofo!
This was followed by a package he had been working on. I wouldn't say it was the greatest, but it wasn't all bad either.

Me:
goodness gracious.

"DUDE! SQL IS A PILE OF SHIT FOR EXPRESSING APPLICATION LOGIC!!!"

disagree (naturally). It's incredibly easy to do, you just don't know how yet...and it seems even the Oracle professionals out there don't either.

I'll tell you this, the crazier the SQL or PL/SQL needed to derive and manipulate data the poorer the design. Start with a good design and it all becomes very simple.
Of course note the use of "naturally" in my lexicon. Thanks Jake.

Mr. M:
well dude, we are back to our old discussion - you arguing that procedural sql code is perfectly fine for building apps, and by extension, that the last 20 years of computer science have basically been a misguided lost journey down the meandering, fruitless trail of oop. um.....no. select this from that. otherwise keep that sql crap caged up where it belongs.

btw, do the db guys here suck? seriously. i'm not competent enough to judge. (to be fair, apparently that crap i sent you is fairly old stuff, but still....)
Me:
I would say, based on limited knowledge of software development, that the OOP movement was started because the database (specifically Oracle) was not mature enough to do what was needed. Plus, I seem to recall that the OOP movement was supposed to have solved all the world's problems by now.

It's further propogated due to the needs you specified in our discussion that day at WellCare (i.e. performance). I still believe that if application developers better understood how a DB works (Oracle, MySQL, SQLServer, etc) they would end up writing less code. Database constraints alone force you to write less (and better) code simultaneously ensuring good data.

The code that I did look at (first 1000 lines or so) isn't great.
1. With all those IF THEN ELSE statements it's telling me that there's probably a better way to store the data. Looks like they're missing an attribute that should be contained with a track.
2. using Object (PL/SQL) types to store data in the way they seem to be doing it is not the best way. Again, probably a design issue.
3. When you do something like this:

UPDATE pb_album_metadata
SET primary_digital_flag = 0
WHERE album_id IN (SELECT b.album_id
FROM (SELECT a.album_id AS album_id,
MAX(a.album_id) OVER (PARTITION BY a.standard_upc) AS latest_album_id
FROM pb_album_metadata a
WHERE a.standard_upc = g_album_tab(1).standard_upc ) b
WHERE b.album_id <> b.latest_album_id )
AND primary_digital_flag <> 0;

They should probably have considered end_date as an attribute of the album metadata. While analytic functions are pretty cool, they're more for analyzing (OLAP) and not OLTP environments.

That's for starters and without table definitions...
Me (again):
oh yeah...and PL/SQL is/was built on top of ADA, FYI.
Mr. M:
"I still believe that if application developers better understood how a DB works (Oracle, MySQL, SQLServer, etc) they would end up writing less code. Database constraints alone force you to write less (and better) code simultaneously ensuring good data."

Huh? What are we not understanding? What would be an example of a constraint that would force us to write less and better code?
Me:
CHECK, NOT NULL (same as CHECK) and FOREIGN KEY constraints all fit into that category.

Instead of having to check if a value is NULL when submitting it to the database, just submit and let the database throw an error, which you'll conveniently capture in your Exception block. Same goes for CHECK constraints, columnA should be either one of three values (test, test1, test3), you won't have to check that in your application code, just catch the exception. FKs insure good data (proper relationships).

A different perspective. If you are going to pay for it, use that damn thing. If you don't care, don't do it. But over the course of an "enterprisey" application, you'll end up writing more code and make maintenance that much more difficult (did I do it there?). Just those constraints will force you and the teams of application developers to put good data into the database.

You can still do it in the application of course (form validation is a good place)...
Mr. M:
Ahh, jeez dude, I wasn't sure if you were referring to the literal "constraint" or not.

Dude, even f_cksticks like redacted I think have a decent enough understanding of when and how to use db constraints. It's when you get into things like cursors or cost plans of subselects and anonymous tables (i think that's the name for it - where I say select something from (select some other crap). Then we defer to db gurus like yourself.

But dude....."you won't have to check that in your application code, just catch the exception".......uh, don't ever repeat that off our little email group. And definitely don't go posting that on your blog. F_ck me man, it's a damn good thing we keep you db folks caged up in that rdbms box....
Me:
So we've had this discussion at work...for a high transaction system, do Java/C/etc handle exceptions well or what?

Why is it bad to deal with exceptions rather than coding to avoid them?

I highly doubt even redacted understood database constraints...him and his cohorts believed all database software would be commoditized and MySQL would be king in short order.
Mr. M:
"for a high transaction system"

Or for any system really....

To use your example of check constraints (is this value one of....) or not null constraints, checking these rules in the Java code and in the database code would seem to violate DRY. But we do that alot, and it is acceptable in certain cases. For instance, we also probably violate DRY if we're checking this same rule say in Javascript on the front end. But we accept this tiny violation of DRY because it tends to make for a better user experience and as a performance gain too, for we avoid the round trip to the server. Now, what your advocating here is close to the same thing. You're basically saying, don't check for the not null constraint in Java code, just go ahead and hit the database, let the database throw back an exception to the Java code, presumably correct the problem, and then make another roundtrip to the database again. Dude, what are you thinking?!? This to say nothing of the fact that this also could be considered a violation of Fail Fast, and a violation of Defensive Programming - what happens if the dba forgot to add the not null constraint in production?

Dude, listen to this guy. For a "high transaction system" basically you want to treat the database, as much as you can, as just a dumb data holder. A f_cking dumpster, that you just throw sh_t into and pull shit back out, using no vendor-specific features if at all possible.

Again, for we've had this discussion, but even in everyday apps, not just on Wall Street, the database is the bottleneck. And the database is the hardest layer to scale. So given those facts, you should design your app to do as little work in the database as possible.
I was laughing at this point because the link above points to one of our consulting architects (I'm not really sure what his role is at this point).
Me:
i agree in any application that you want to minimize the number of round trips...

shocker...he's one of our architects. he's spot on in many instances, but...

database is the bottleneck because people don't know how to write SQL. I'll certainly concede the wall street applications (for the time being anyway), but the rest of us with what we do on a daily basis...Oracle will scale far beyond the demands they place. When that bottleneck shows up, 9 times out of 10 some dumb-ass c#/java guy thought he could write a better query than i. besides, what's the idiot doing anything but a simple join anyway? probably poor data model to start with...and we're right back where we started (sort of).
Mr. M:
"database is the bottleneck because people don't know how to write SQL.....some dumb-ass c#/java guy thought he could write a better query than i."

Dude, I'll grant you, people don't know how to write SQL, myself included. But that's not always why the database is the bottleneck. I think it's safe to say that's not even the majority of the reason. Yes, there are apps written by people who were just idiots, and they needlessly pummel the database, but that's definitely not the majority of scenarios. At my work the database is the bottleneck, and we run RAC in production. It's my understanding that even with RAC, there is a limit to how much you can scale that out. But any layer up from the database we are basically unlimited in how much we can scale that out. So it's very easy to stick another Apache box in front, or fire up another Weblogic server. But I can't do that with the database. We have 24 Weblogic servers for redacted. The database is the bottleneck. And we don't have shitty sql code in the app. In fact, we have very few hand-written queries anywhere in the app. Persisting something to a database is really a low-level concern that as an application developer I shouldn't even have to be bothered with, except for the rare corner cases where the persistence abstraction I'm using is just a little bit too generic to handle things effectively. And we don't use these ORMs because we don't know how to write sql. To effectively use an ORM tool requires a deeper understanding of sql and databases than simply being able to write solid SQL code. (Not saying Java devs who use ORMs know SQL better than a dba, just that it requires better sql skills than is required of a Java dev to simply write JDBC/SQL.) Now, before you try to tell me that my ORM library isn't able to write as efficient of sql code as a dba of your caliber, keep in mind that ORM tools are pretty advanced. They're able to intelligently do things like batch sql updates, and let you apply transactional semantics much easier than with raw jdbc/sql. But the overwhelming reason developers have so thoroughly adopted ORM is because Structured Query Language is such a nasty piece of shit for expressing application logic. SQL is a declarative, procedural language. It's totally unsuited for writing application logic! This, more than anything else, is why organizations and dev teams should seek to restrict what is handled within a relational database as much as possible - because the programming interface to it is a fucking ancient backward dying dinosaur.
Mr. V (note, not Mr. M):
My 2 canadian cents:
The polyglot approach "... use different languages for different domain ..."
Database was developed to manipulate data and should remain there.
General purpose language was developed to encapsulate logic and should remain in that domain.
You should not use DB to encapsulate business logic (in my opinion) no more than you would use HTML to create complex logic.

While Java, C#, Python, etc are described as General Purpose languages, they, too, are really domain-constrained. Their expressiveness are confined (and should be) to express low-level, tersed, explicit, verbose, and repetive logic (if that makes any sense). Languages such as these are more suitable for low-level abstraction on top of which, richer, more expressive languages can be built. We are seeing this now with the emergence of languages on the JVM (and arguably on the .Net's CLR).

I think SQL as we know will take a back seat and a smart developer somewhere will create a new domain-specific language that lives on the VM and will push the SQL expression out of the RDBMS and closer to the code that needs it. We are not there yet, but Microsfot is trying (see LINQ and all ORM papers). This is not to say that there won't be isntances where tuning in the SQL-native language won't be necessary. However, 80 to 90% of simple CRUD cases will be handled closer to the logic code that uses the data code.

Again, that's my 2 canadian cents... I could go on. But I have a meeting with redacted.
Mr. V, I believe, is a little bit more sensible. Mr. M on the other hand is just trying to rile (sp?) me up.

Me:
Someone will probably create something like that, but it still gets at the heart of one of my arguments, many developers don't know how to use a database thus will go to any means to circumvent it. Embrace it I say.

Ultimately for me, it comes down to simplicity. I can write an application using PL/SQL that will scale very well for 90% of the solutions. Of course that doesn't include the "fancy" javascript/Ajax stuff needed for the UI. In my ever so humble opinion, this is a much better solution for a business in the long run.
1. You're more likely to change the middle tier than the database tier. Java, asp, Ruby, what's next?
2. Fewer moving parts means lower cost of ownership, even with the CPU costs. One person, one, was able to build and maintain a 350 page website. Be hardpressed to do that with the more expressive languages.

I think all of us are purists and very passionate about what we do. It's probably the main reason we get along. I thoroughly enjoy these conversations because it does force me to think...and that's always a good thing.
Mr. V:
Haaa chet.
You sorta made my point than diverged away. Building an app in all PL/SQL is dangerous. It's no safer than me building an app in all Java. I can build very limited app in all Java. The moment I need to interact with other domain (UI, data, low-level native code, etc), I have to switch to something that is closer to what I am trying to do. If I need to create UI, I will pick a ui-centric environment, when I need to talk to DB, I will pass connect to a db and send in my SQL, and so forth. I will use Java as the glue to bring it all togher.

In the end, we may be saying the same thing, but using different accent. O well.
And that's where it ended. I must say it's always fun. Mr. M and Mr. V are both very smart individuals and I highly respect what they do. We have different perspectives...but I think they listen, if only a little, as I listen to them. Their voices creep up on me especially now...which is definitely a good thing.

23 comments:

Crisatunity said...

Chet, of course you are more correct more reasoned and better equipped to carry your points to light, but you are a fool to take the bait on this boring tired old fight with a tool that gets off on razzing you.

oraclenerd said...

@crisatunity

He does do that doesn't he? I always forget he's just trying to rile me up.

The first time it happened I was mad...now I just laugh. I know he doesn't truly believe it to that extreme (at least I think).

He's well known for his antics...

Noons said...

"Persisting something to a database is really a low-level concern that as an application developer I shouldn't even have to be bothered with"


Yikes! I thought I had heard all possible variations of idiocy and sheer unadulterated ignorance, but it looks like the oop/java brigade continues to excel at it...

mcohen01 said...

@Noons

"Persisting something to a database is really a low-level concern that as an application developer I shouldn't even have to be bothered with"

What exactly do you think is "idiotic and ignorant" about this statement? As an application developer, when I write something to the database all I really care about is that this particular bit of data is kept around somewhere so that I can look at and do something with it later on down the road, even if I turn off the power on the computer. But I'm not especially concerned with the fact that the means of achieving this goal is via a relational database. In fact, to be quite clear about it, I don't give a sh_t at all that we're using a relational database. It's an implementation detail that in a perfect world I really shouldn't be concerned with. So for example, it shouldn't really matter to me that we're using an Oracle database over say, an object database or xml files, or Amazon S3. We're not quite there yet, but we're making progress. You can examples of this in ORM technology. So instead of writing "insert into...." I just call save() on an object. If I want the data back again I just call get(id) and I pass in some identifier. This happens to be a primary key but it could be any identifier really. Already if I'm using an ORM the actual brand of database I'm using is transparent to me. I could swap Oracle out for Sybase with a configuration parameter.

See, the real problem here, and again, on a site like oraclenerd.com, it's clear this is not a popular opinion, but the real problem here is that all you guys like Chet make your bread and butter programming Oracle databases. If Oracle's not on top, and especially if rdbms's lose the stature they've traditionally held within the enterprise, this totally threatens your livelihood because of how personally invested you are in the technology. That's why Chet, and I suspect most folks here, are totally standoffish if not outright opposed to things like MySQL or datagrid-type products like Coherence.

The real problem here is that you db guys basically have a lower level view than that of an application developer. As an app dev you're concerned with the entire application, and as such you're constantly working at the database layer as well as the application layer. So you have this holistic view of things. Whereas dba's have a much more narrow view of things. So if we extrapolate current trends outward, "the network is the computer," we can see a world where applications use multiple distributed heterogenous datasources located in the web ether, the cloud to use current fashionable terms, and quite possibly require no actual local disk-based persistence, but in the rare event that such persistence is required it's achieved with the lingua franca of the web, XML, and there simply is no longer a compelling reason to use expensive legacy decaying dinosaurs like an Oracle rdbms. You should take a page from the Agilists and instead of resisting change embrace it, for this is future of things.

Narendra said...

And if I remember correctly, few years ago(around Y2K), there was a belief JAVA and Object-Oriented Programming will outsmart all other technologies (including databases) and make everything obsolete. So JAVA/OO is the way forward...
Well, we are in 2008 & we still have databases (Oracle, SQL Server, Sybase etc.) and all large/medum/small organisations use it extensively.
As the famous Tom Kyte says..."Applications come and go, data remains"
Let us see 5/10 years from now, how many businesses "believe" in this "database-are-data dumps" approach.

John Scott said...

@mcohen01

>I could swap Oracle out for Sybase >with a configuration parameter.

Good luck with that.

I've seen plenty of projects that have 'database independence', very few of them (none maybe?) have ever successfully switched out one DB for another just with a 'configuration parameter'. It's a nice ideal...rarely holds up in practice though.

Whether you use Oracle, Sybase, MySQL etc you *should* be using all the features you have "paid for" (where paid for means in terms of money or time investment), rather than aiming for the lowest common denominator of functionality.

John.

mcohen01 said...

@John Scott

Agreed. True database independence is rarely achieved, and the actual requirement for it is probably even rarer. I was just using this as an example of abstraction away from the database. So it's possible right now to abstract away the specific details of the database we're using. Soon it will be possible to abstract away from the developer the very fact that we're using an rdbms at all for persistence. This is a Good Thing. And a natural progression in the kinds of abstractions that we work with every day as developers.

@Narenda

"And if I remember correctly, few years ago(around Y2K), there was a belief JAVA and Object-Oriented Programming will outsmart all other technologies (including databases) and make everything obsolete."

I'm not sure if anyone expected a general purpose language like Java to replace rdbms's. I certainly don't know of anyone who was espousing such an opinion. It is hard however to overstate the dominance Java has achieved in about a dozen short years. Especially when you consider the very strong desire of Microsoft to bring forward in the marketplace opposition. Considering Microsoft's considerable clout in the industry and with developers, .NET has largely been a failure.

"So JAVA/OO is the way forward..."

Well, Java really has been a way forward in that it's become the defacto standard for your average corporate developer to work with an OO language while enjoying the productivity boon provided by automatic memory management all provided on a robust scalable platform. As they say, Java is the COBOL of the 21st century.

"Well, we are in 2008 & we still have databases (Oracle, SQL Server, Sybase etc.) and all large/medum/small organisations use it extensively."

This is true. But the writing is on the wall. Datagrid products like Tangosol Coherence are still somewhat new, but very soon that space too will become commoditized. And it's not just Wall Street trading applications that can realize large benefits from these datagrids. Plus, as I said before, the accelerating trend is toward distributed datasources located elsewhere on the network (e.g. the web). The importance and primacy of position of the rdbms in the enterprise is clearly waning. Quite soon I imagine these dinosaurs will occupy only a small, niche space in organizations.

"As the famous Tom Kyte says...'Applications come and go, data remains"'"

This is largely correct. But don't confuse databases with "data." They are most certainly not the same. Remember, what we're talking about here is simply the specific mechanism of data storage, one that is rather long in the tooth, and showing it's age, and is quickly becoming unsuited for modern software development given current trends.

John Scott said...

@mcohen

"Soon it will be possible to abstract away from the developer the very fact that we're using an rdbms at all for persistence. This is a Good Thing."

It has been possible to do that for a very long time (through the use of library functions, API's etc). I worked on a project nearly 20 years ago where the developers didn't have direct access to the DB we just used common API routines which handled it all for us.

I am not against frameworks/persistence layers etc per-se, *as long* as they are well written and understand the strengths and weaknesses of the underlying database. It is when these tools try to be too generic and fail to take into account the subtle ways that different vendors databases can work when chaos reigns.


"what we're talking about here is simply the specific mechanism of data storage, one that is rather long in the tooth, and showing it's age, and is quickly becoming unsuited for modern software development given current trends."

You mean unsuited to the way you want to use it? Don't generalize that *everyone* wants to abstract in this way, because not everyone does.

Perhaps the reason you're finding it unsuited is because you're trying to fight against it rather than working with it?

mcohen01 said...

@John Scott

"It has been possible to do that for a very long time"

What I'm talking about here is what the JPA folks sometimes offered as their goal - generic, pluggable persistence. So I could swap out a SQL provider for an XML db provider or an Amazon S3 provider. We're not there yet, but this is the trend.

"You mean unsuited to the way you want to use it? Don't generalize that *everyone* wants to abstract in this way, because not everyone does."

Well John, the adoption of ORM technologies, pretty much across the language spectrum, suggests that I'm far from the only one who finds an SQL programming model lacking. And there is good reason for this. The relational model is excellent for data storage, and SQL is very good at set-based data retrieval and manipulation, but SQL just can't compare to the power and expressiveness of modern programming languages. Further, OO techniques have proven to be quite useful at managing complexity within application code, but the disconnect between the object world and the relational world is so strong that tremendous effort has been devoted toward relieving developers of the onerous task of transitioning between the two worlds, with commensurate gains in developer productivity.

"Perhaps the reason you're finding it unsuited is because you're trying to fight against it rather than working with it?"

It's not a question of fighting with it or working with it. The point is that you shouldn't even have to work with it. It's a low-level concern that developers shouldn't have to deal with. I have this bit of data and I want to save it, and later on at some point in the future I want to work with it again. That should be the extent of my concern. It's analagous to manual memory management. I just want to create an object, I shouldn't have to be concerned with the details of allocating memory in order to do that. And as Java has superceded C, soon enough the low-level programming model of the rdbms will be superceded.

John Scott said...

@mcohen

"So I could swap out a SQL provider for an XML db provider or an Amazon S3 provider. We're not there yet, but this is the trend."

What is the trend? That people want it? Sure...people always want things that sound cool. Does everyone *need* that level of abstraction? Almost certainly not.

You're talking about a small *niche* area of applications there, the ones were it makes sense to switch out a SQL Provider to an XML Provider etc.

"the adoption of ORM technologies, pretty much across the language spectrum"

Don't forget, your perception of how much ORM is used is based on your own experience (I'm guessing it's the area you work in on a day to day basis). There are lots of folks/companies/languages out there that don't use/require/enforce ORM to be useful.

"but SQL just can't compare to the power and expressiveness of modern programming languages."

Horses for courses as we say over here in the UK. Nobody is pretending that developing a program using PL/SQL is the same as using C (for example). Neither is inherently 'better' than the other, it depends very much on the context of what you're trying to achieve.

"Further, OO techniques have proven to be quite useful at managing complexity within application code"

Sure, I agree, they can. They can also be (ab)used to *create* additional complexity. So care is required not to over-engineer the solution.

By the way, you know that Oracle supports Object types in the database right (and has done for some time - they may not be perfect yet but they're there)?

"It's a low-level concern that developers shouldn't have to deal with"

It depends what level you're talking about.

Should I, as a developer, care which sector of the disk my data is stored on? Nope, probably almost certainly not.

Should I, as a developer, know how the Employee table relates to the Department table and the Salarys table? Almost certainly yes (assuming that's the data I'm working with).

Without knowing how my data is organized within the database, how can I possibly make best use of that data when it comes to processing it?

If we take this to the natural conclusion, you are essentially saying that given 2 developers -

1) Developer X who understands the development tool.
2) Developer Y who understands the development tool AND the database (including the data).

You are saying that Developer X has the 'edge'? I couldn't disagree with that more.

John

Mike said...

I'm not nearly as versed in DB matters as most of you, and I'm not a biochemist, but let me say what I believe: Nothing lives without heat. If this is true, I would say it applies to ideas as well. So with that, let me say "thanks" to Mr. M for being the OHS (official heat source) on this post. Keep the language hot, that's what I say; it keeps the subject alive. Just guard yourself and don't take any of it personally (as I can see Chet has mastered).
(So philosophy and general theory is my thing you see.)
Now I stumble into the database arena. (Disclaimer: This could be ugly.)
What the fuck, Chet?!! jkjk...that was me imitating Mr. M. (Maybe I can be the OHS some day.)
Another theory: chaos rules. If you like the database-less database, make it work. Promote it. Perfect it. Database-centric type? Same thing. They both work.
What will rule depends not so much on what's *best* as it does which side works harder to make their side supreme.
What demands does the future hold for the use of data and what will the application software trends be? Anyone? Nostradamus here today? Yeah, nobody knows though many of you know in what direction things appear to be heading at the moment.
Here's what we do know: They both work exceptionally well when the people using them know what the f**ck they're doing (nod to Mr. M).
Let's admit: When you're up against Oracle, you're facing a worthy opponent and it really doesn't matter what the "facts" of the matter are.
Want to unseat the giant? (We all love to see a giant fall.)
For me, for now, I'll continue trying to master the ways of the giant so when I expose its weaknesses I'll know what the f**ck I'm talking about (Mr. M).

mcohen01 said...

"What is the trend? That people want it?"

No, the trend is increasing levels of abstraction. This is simply the history of computer programming.


"Don't forget, your perception of how much ORM is used is based on your own experience (I'm guessing it's the area you work in on a day to day basis)."

I work with ORM tools probably on a daily basis. My perception isn't based on my personal experience. Even a cursory glance at the landscape reveals widespread adoption of ORM technologies.
http://en.wikipedia.org/wiki/List_of_object-relational_mapping_software


"There are lots of folks/companies/languages out there that don't use/require/enforce ORM to be useful."

And nowhere did I claim such a thing.


"Neither is inherently 'better' than the other, it depends very much on the context of what you're trying to achieve."

This stems from Chet's original comment that "PL-SQL was an excellent choice as an application development language." I'm arguing that it most definitely is not. It's suited for the rather narrow concern data manipulation within an rdbms.


"They can also be (ab)used to *create* additional complexity. So care is required not to over-engineer the solution."

Yes, anything can be misused or abused. What's the point? On balance, OO has been a boon for developers. I don't think that's a controversial statement.


"Should I, as a developer, know how the Employee table relates to the Department table and the Salarys table? Almost certainly yes (assuming that's the data I'm working with). Without knowing how my data is organized within the database, how can I possibly make best use of that data when it comes to processing it?"

No, you're concerned with how an Employee relates to a Department and to a Salary. These are business concepts, which quite often have rich functionality and interactions among them that must be expressed in the application. The last 20 years of software development has proven that these relationships and behavior are best modeled using OO techniques. The fact there are tables involved in an rdbms should only be an incidental implementation detail. If the data were persisted to disk in xml files it should make no difference to me. The level of myopia with you database guys is staggering. Expand your horizons. Broaden your perspective. Data and the database are not synonymous.


"If we take this to the natural conclusion, you are essentially saying that given 2 developers -

1) Developer X who understands the development tool.
2) Developer Y who understands the development tool AND the database (including the data).

You are saying that Developer X has the 'edge'?"

Nice straw man. Of cours I'm not saying that. It's not an argument to simply say I don't understand the database. If I were to make a comparable argument I'd suggest you database guys don't understand OO. But I'm not suggesting that. I'm giving you the benefit of the doubt. I would expect to be afforded the same courtesy. Most devs I've come across who share my views that SQL is poorly suited for expressing application logic and look to ORMs to bridge the gap between the relational and object worlds have excellent knowledge of database technology.

John Scott said...

@mcohen

"Even a cursory glance at the landscape reveals widespread adoption of ORM technologies."

'Widespread adoption' (whatever % that translates into) is not necessarily the same thing as 'successful adoption' sadly ;)

"This stems from Chet's original comment that "PL-SQL was an excellent choice as an application development language. I'm arguing that it most definitely is not. It's suited for the rather narrow concern data manipulation within an rdbms."

Reality disagrees with you, there are many applications running out in production with a strong PL/SQL code base.

"If the data were persisted to disk in xml files it should make no difference to me."

Depends if you take the very narrow focus view of "All I care about is my code" rather than the "I care about the application as a whole".

As a developer I think you should care about aspects of the application that you may not be directly involved with.


"The level of myopia with you database guys is staggering. Expand your horizons. Broaden your perspective. Data and the database are not synonymous."

Hmm, aren't you conflicting with your prior statement there? On the one hand you're saying you don't want to know about such (DB) details, on the other you're telling *us* to broaden our perspectives...pot...kettle ;)

"If I were to make a comparable argument I'd suggest you database guys don't understand OO."

Again...those generalization will really kill your argument for you. FYI one of my favourite development tools (outside of Oracle) that I've used is Delphi, which I used for many years and I'm certainly more than familiar with OO development.

Actually, I consider myself a Developer/DBA (or DBA/Developer depending on the day of the week), so I have seen things from both sides of the fence. That's why I like my logic inside the DB, I've seen what putting it outside the DB can do when you change your development tool/infrastructure ;)

mcohen01 said...

'Widespread adoption' (whatever % that translates into) is not necessarily the same thing as 'successful adoption' sadly ;)

So now you're going to argue that all these developers using ORMs are failing with them? C'mon dude. Your point's irrelevant anyway. I used the broad adoption of ORM technologies to illustrate the point that developers are unsatisfied with the programming model offered by SQL and would prefer to use OO techniques.


"Reality disagrees with you, there are many applications running out in production with a strong PL/SQL code base."

No John, reality doesn't disagree with me. We're talking about the suitability of SQL as an application development language, not the possibility. You can write applications in assembly too, that doesn't mean it's a good idea.


"Depends if you take the very narrow focus view of "All I care about is my code" rather than the "I care about the application as a whole". As a developer I think you should care about aspects of the application that you may not be directly involved with."

I should be concerned with implementing business logic. Data persistence is a relatively low level concern that really shouldn't occupy my time.


"Hmm, aren't you conflicting with your prior statement there? On the one hand you're saying you don't want to know about such (DB) details, on the other you're telling *us* to broaden our perspectives...pot...kettle ;)"

Is this really the best you can do? I shouldn't be concerned with the details of the persistence mechanism, but I'm fully aware of the available mechanisms. Yes, you db guys need to broaden your perspectives. Continually I see you making the mistake of confusing the more general concept of data with a database.


"If I were to make a comparable argument I'd suggest you database guys don't understand OO."

Again...those generalization will really kill your argument for you.

That's why I'm not making them John. Reading comprehension.


"Actually, I consider myself a Developer/DBA (or DBA/Developer depending on the day of the week), so I have seen things from both sides of the fence. That's why I like my logic inside the DB, I've seen what putting it outside the DB can do when you change your development tool/infrastructure ;) "

Or, applications come and go, the data always remains. But the development platform changes typically as new applications come and go. Implicit in that is new application functionality. The fact of the matter is, there's not much room anymore for application logic implemented within an rdbms. The web dominates today, and special purpose environments like RoR or Grails provide developers with unparalled productivity. CRUD apps go up in minutes. Hand coding SQL statements is quickly becoming a thing of the past. Get used to it.

John Scott said...

@mcohen

"So now you're going to argue that all these developers using ORMs are failing with them? "

Where did I say that? I said that 'widespread adoption' is not the same as 'successful adoption'.

Just as you're not saying (I hope) that they're all successful, I am saying that they're not all successful.

"We're talking about the suitability of SQL as an application development language, not the possibility."

PL/SQL not SQL...

"Continually I see you making the mistake of confusing the more general concept of data with a database."

We're not confusing it, what we are saying is that when you store your *data* in a *database* it offers many advantages and features over (for example) storing it in an XML file on the filesystem, surely you concede that?

"Hand coding SQL statements is quickly becoming a thing of the past. Get used to it."

Ummm...you're really mistaking me for someone else I think?

My tool of choice for development these days is Oracle Application Express (http://otn.oracle.com/apex).

It's a web development tool, runs completely inside the database. Makes a lot of web development tasks extremely simple (such as session handling etc), CRUD applications are a breeze.

The difference is, I can write my SQL and PL/SQL inside APEX because I'm comfortable at that level, I know how to code SQL and PL/SQL. The tool works for me (it's not going to work for everyone, just like Java/.NET etc isn't).

My point is, with the tool I use, PL/SQL becomes a very real development environment (because under the hood that is what APEX is using).

mcohen01 said...

"Where did I say that? I said that 'widespread adoption' is not the same as 'successful adoption'. Just as you're not saying (I hope) that they're all successful, I am saying that they're not all successful."

Their success or failure is irrelevant to our discussion. The important point was that ORM's adoption has been widespread.


"PL/SQL not SQL..."

Either one.


"We're not confusing it, what we are saying is that when you store your *data* in a *database* it offers many advantages and features over (for example) storing it in an XML file on the filesystem, surely you concede that?"

In some cases an rdbms offers advantages, in other cases an xml db, in still yet other cases something like Amazon S3 is the right choice. The cases where rdbms is the right choice are shrinking. Regardless, this is low-level implementation detail that I shouldn't be concerned with, just as manual memory management is a low-level implementation detail that I shouldn't be concerned with. Really, why is this so contraversial for you?


"The difference is, I can write my SQL and PL/SQL inside APEX because I'm comfortable at that level, I know how to code SQL and PL/SQL. The tool works for me (it's not going to work for everyone, just like Java/.NET etc isn't). My point is, with the tool I use, PL/SQL becomes a very real development environment (because under the hood that is what APEX is using)."

You may be comfortable using some tool, but the fact remains that PL-SQL simply cannot match the power and expressiveness of today's OO languages, and as such, should be avoided for general purpose application development. This was Chet's original comment that started all this.

Chet's comment:
"It is a great platform if you know how to leverage it. PL/SQL is a 3GL (or is it 4) and you can write entire applications using it...and if you do that right, an incredibly robust and scalable application at that."

John Scott said...

"Regardless, this is low-level implementation detail that I shouldn't be concerned with, just as manual memory management is a low-level implementation detail that I shouldn't be concerned with."

But again, I'd disagree...I think all developers should have a good grounding in memory management, it's a fundamental part of knowing how computer systems operate.

" but the fact remains that PL-SQL simply cannot match the power and expressiveness of today's OO languages, and as such, should be avoided for general purpose application development. "

Surely you're not suggesting that *every* development has to be done using OO techniques? Surely not...you mean that we should be coding using OO techniques whether the situation warrants it or not, just for the sake of using OO?

Besides..should you want to do that, I refer back to my earlier comment, Oracle PL/SQL has object support built in (for some years as I said), so I can use OO techniques with PL/SQL should I want to (whether I would or not is a different matter).

So, I still fail to be convinced that I shouldn't be using PL/SQL for 'general purpose development' (I think you also worded that incorrectly, surely you mean PL/SQL should be avoided for 'OO development', since I would put 'general purpose development' into the non-OO category).

mcohen01 said...

"But again, I'd disagree...I think all developers should have a good grounding in memory management, it's a fundamental part of knowing how computer systems operate."

That's what university is for. I know how registers work, that doesn't mean I should program against them when writing applications. Again, the history of computer programming, in other words the history of higher and higher levels of abstraction, bears this out.


"Surely you're not suggesting that *every* development has to be done using OO techniques? Surely not...you mean that we should be coding using OO techniques whether the situation warrants it or not, just for the sake of using OO?"

No, I'm talking about application development. If you've got some batch job that just has to push data around from one table to another, or from one db to another, for whatever reason, then sure, use pl-sql for that. But again, to go back to Chet's original statement, "PL/SQL is a 3GL (or is it 4) and you can write entire applications using it...and if you do that right, an incredibly robust and scalable application at that." Entire applications should not be written in pl-sql.


"Besides..should you want to do that, I refer back to my earlier comment, Oracle PL/SQL has object support built in (for some years as I said), so I can use OO techniques with PL/SQL should I want to (whether I would or not is a different matter)."

Oracle or Postrges would be a rather poor choice. Oracle objects aren't really in the same class as OO languages and environments like Java or C#. It's not even really a valid comparison.



"No, I still fail to be convinced that I shouldn't be using PL/SQL for 'general purpose development' (I think you also worded that incorrectly, surely you mean PL/SQL should be avoided for 'OO development', since I would put 'general purpose development' into the non-OO category). "

No, I chose my words carefully. PL-SQL should only really be used for strictly database development, it should be avoided for general purpose development.

mcohen01 said...

Well Chet, i would say that you and your readers haven't been able to defend your case very well. (And really, how could you be expected to?) I'm sure it's not a pleasant prospect, but the reality is that the RDBMS is increasingly becoming irrelevant. You and your readers have a large personal investment in the technology, which you stand to lose to the degree it does become relegated to a niche market. But it does you no good to bury your head in the sand and deny the reality. Instead you should embrace the reality of software development today and get on with the task of learning more current technologies.

Crisatunity said...

Let me wrap this up in a bow. If you think there is, in fact, a "vs." in any portion of either the application code platform or the persistent repository platform in any way - you are tool that every one should ignore.

The supposition of the "vs." even existing is based on FUD by those familiar with only a narrow understanding of the many techniques to get the job done.

John Scott said...

@mcohen

"And as Java has superceded C"

By the way, I think you left your credibility back there...

"No, I'm talking about application development."

So was I.

"Entire applications should not be written in pl-sql. "

Why not? You've failed to come up with a compelling argument so far, other than "they shouldn't" or that "PL/SQL doesn't have as good OO capabilities as Java" (my wording not yours).

Nobody disagree's with you that Java (and some other languages) has much better OO capabiities. However that's hardly the point is it?

If i don't want to develop using an OO methodology (and once again, remember I can do that with Oracle if I wanted to) then PL/SQL is a perfectly suited development tool for many requirements.

"Oracle objects aren't really in the same class as OO languages and environments like Java or C#."

Nice pun by the way...

Oracle Objects have attributes, methods, inheritance, supertypes, subtypes, polymorphism and lots of other features.

Which critical features do you believe they're missing that makes them so much less useful than other OO tools have?

"it should be avoided for general purpose development."

Why? Because you say so? Again, you have failed to come up with a compelling argument why it shouldn't be used. If your entire argument hangs against it not being 'OO enough' then it sounds like you're too wrapped up in the concepts to see the bigger picture.

Anonymous said...

select count(*) from companiesIhaveworked where oracle = "2 much $"

result - all. ;)

mcohen01 said...

or:

select count(*) from companiesIhaveworked where postgresql = "good enough" or mysql = "good enough"

result = all

or better yet:

select count(*) from companies99%ofDevsHaveWorkedAt where postgresql = "good enough" or mysql = "good enough"

result = all

Of course, this assumes that Oracle somehow buys you something that you don't get with a PostgreSQL. Beyond the gimmicks like AQ, I'm not sure that's a valid assumption.