tag:blogger.com,1999:blog-8884584404576003487.post2260666184695354690..comments2024-02-29T09:43:12.251-05:00Comments on ORACLENERD: How Do You Normalize a Tweet?oraclenerdhttp://www.blogger.com/profile/12412013306950057961noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-8884584404576003487.post-36597392699307846662009-12-26T15:23:16.760-05:002009-12-26T15:23:16.760-05:00"The point is, the relational model is rapidl..."The point is, the relational model is rapidly giving way to other forms of structuring data."<br />So ? Whether you are using Relational, Dimensional, XML, Key-Value or even find a way of getting Object Oriented databases to actually work, the same questions need to be answered. What are the identifiers, what are the attributes, what are the dependencies, what can be changed, how do you search for data, how do you write it. Seen it, done it.<br /><br />The data-modelling task changes only a small amount for implementation details (though would be a lot harder for key-value because it has to anticipate more at the design stage). If you think any of these issues goes away just because it is or can be held in RAM, you need to read up on the details.SydOraclehttps://www.blogger.com/profile/08828771074492585943noreply@blogger.comtag:blogger.com,1999:blog-8884584404576003487.post-58228404043136863292009-12-24T09:58:12.179-05:002009-12-24T09:58:12.179-05:00"While I believe you are correct that, in the..."While I believe you are correct that, in the future, more data will be held non-relationally, that doesn't mean there will be less data held in relational form. It's just that more data will be held in total."<br /><br />The total number of bits on disk arranged according to the relational model may be lesser or greater, but the fact is that the relational model is already becoming obsolete. As with all trends in technology, the pace of this retirement will only accelerate.<br /><br />"Formal, automatic, data processing will require more structured data."<br /><br />Data processing is perfectly feasible with unstructured data, it's just typically easier with structured data. Of course, "structured data" takes many forms. The point is, the relational model is rapidly giving way to other forms of structuring data.<br /><br />"PS. If you believe every tweet ever sent is in RAM, I suspect you are very wide of the mark. I'd be surprised if a high percentage of tweets more than a day old are anywhere except on disk."<br /><br />I'm not sure why you would say that. Saying it's in RAM doesn't mean everything exists in RAM on every node in the cluster. There are plenty of systems with everything in RAM that have a lot more data than every tweet ever sent.mcohen01https://www.blogger.com/profile/10754310184888464447noreply@blogger.comtag:blogger.com,1999:blog-8884584404576003487.post-46517176350329393872009-12-24T07:38:26.102-05:002009-12-24T07:38:26.102-05:00That was the point. The relational model was devis...That was the point. The relational model was devised for repetitively structured data, that would be processed automatically. Tweets are less structured and processed 'informally' by people.<br /><br />While I believe you are correct that, in the future, more data will be held non-relationally, that doesn't mean there will be less data held in relational form. It's just that more data will be held in total.<br /><br />Formal, automatic, data processing will require more structured data. <br /><br />PS. If you believe every tweet ever sent is in RAM, I suspect you are very wide of the mark. I'd be surprised if a high percentage of tweets more than a day old are anywhere except on disk.SydOraclehttps://www.blogger.com/profile/08828771074492585943noreply@blogger.comtag:blogger.com,1999:blog-8884584404576003487.post-53908930191681603112009-12-23T18:52:49.751-05:002009-12-23T18:52:49.751-05:00How ironic that you chose Twitter for this little ...How ironic that you chose Twitter for this little exploration. It's ironic because Twitter no longer really uses a database, other than as a backup. Everything is in RAM. <br /><br />Probably much quicker than many believe, this paradigm of "data in the cloud" will become the norm rather than the exception. And, for better or for worse, the relational model and RDBMS's will become the modern version of COBOL.mcohen01https://www.blogger.com/profile/10754310184888464447noreply@blogger.com