Everybody talks a different language

When we decided to start building ODE we knew a few things already.  One of those things was that most of our customers already had data warehousing technology.

They had already invested in Microsoft, Oracle, IBM, SAS, Teradata, Informatica or any of the other raft of data warehouse repositories and ELT technologies that are abound in the market place.

We also knew that it would be a big decision on their part to throw out this technology and implement the technology we decided to pick to be able to use ODE and gain the AgileBI benefits that it provides.

Don’t force people to do what they don’t want to

So we decided to develop ODE in a way that meant they could continue to leverage the technology they already had if they wanted to.

This meant we had to be able to deploy ODE on multiple of platforms, leverage multiple data transformation languages and tools, plus read and write to multiple data repositories.

So we had a multi language problem

We though about writing ODE in java. In theory this would allow us to deploy on any platform and any technology.

We knew from experience we would probably end up being forced down a java server path for transformations, and we also knew most of the hard core data warehouse customers we work with would want it to run natively in the data repository they already have invested in.

We knew that there were some language translation tools we might be able use.  Tools where we could write code in say tSQL and it would automagically convert it to PL/SQL.  We tried a couple and our overall experience was “yeah, nah”

We also knew that like our team, our customers would want us to tune any data transformation code to run fast, and this meant being able to tune the code for each technology.  And in fact for specific scenarios within a technology.  For example using columnar store when it was available in the target data repository.

And last but not least we knew maintaining multiple versions of the same product would be a complete nightmare for us.

Enter 3CPO

Our solution was to look at our plan for ODE and work out what we could architect as a standard shared component and what had to be specific to each technology.  We also discovered we would need to manage a couple fo different deployment options as well.

So we ended up with a design we call 3CPO, which stands for:

• Configuration
• Code Generation
• Code Execution
• Orchestration

Config

Config is the relational data model we have built that holds all the configuration required to make ODE run.

This will include definitions of all source and targets, as well as any mappings.  For example

• Raw Vault Hub, Sat and Links
• Business rules to be applied in the business vault
• Measure calculations
• Star schemas to be deployed

It will also include unique options to deploy for a specific environment or to a specific design pattern.  For example:

• Whether to virtualise the dimensional star schemas or persist them.
• End date each satellite record, create tombstone records or do both
• Utilise columnar storage for a satellite
• Create an index

Config is the heart of ODE, without it there is nothing.

Code Gen is the code that builds Code Exec.

It reads the config and generates the code that is needed to create the structures and move the data.
(of course if you select the the virtualised options it will create views that see the data)

Code Gen will come in multiple flavours, so you can deploy it on the technology you already have.

At the moment we are planning to include (overtime):

• Microsoft tSQL
• Oracle PL/SQL
• Oracle ODI
• SAS
• R

There is nothing stopping the wider community building Code Gen and Code Exec for another platform (say Informatica) reusing the code patterns we have already defined.

Code Exec is the code that runs to create the structures or move the data.

(of course if you select the the virtualised options it will be views that see the data)

Code Exec will also come in the same multiple flavours as Code Gen.

Orchestration is the engine that runs the code exec.

It can be run in two design modes.

Engine Mode

Engine mode is when the Code Gen and the Code Exec are executed at the same time.

So effectively ODE will look at the config, create the code exec, execute it, and then rinse and repeat.

Code Deployment Mode

Code Deployment Mode is when Code Gen creates the Code Exec and then you manually promote the Code Exec across your different environments (i.e. Dev > Test > Prod)

Execution

The execution engine that actually tells Code Gen when to build the Code Exec, and also tells Code Exec when to execute will be over to you for a while.

We will deal with it when we have finished building all the features required to support Config, Code Gen and Code exec that will manage the entire process of moving the right data from source to star.

It can be as simple as Cron, or the use of an ETL engine such as SSIS.

Config is the heart of ODE

The configuration component is the core of ODE and the thing that will be maintained as a common component across all developed and deployed versions.

We are maintaining the config component via version controlled  processes, with the code being stored in GIT (as we are with all out code of course).

Our team will fight long and hard as they decide to add a new feature into the config, to ensure ODE doesn’t become bloated but also to ensure we keep adding core features.

We are semi-lingual already

Our core Code Gen and Code Exec is currently written in tSQL.  This is due to both the skills of the people we had available to kick off initial development and the customers we were working with for the initial deployments.

We have also done initial builds in PL/SQL and SAS, but need to move these up to the latest config release.

Hop on the bus

We are not quite ready to open the flood gates and let the world help start adding features to ODE.

We are working on our Test Driven Development (TDD) and Continuos Integration (CI) frameworks at the moment to ensure we can safely test any config and code changes as we add features.  This is core before we can safely start committing contributions.  (not to mention doing the documentation you will need to get started)

But we are keen to talk to anybody who might want to start the journey early with us.

Grab a ticket (they are free) and hop on board.  Its going to be an exciting ride!

C3PO Image src: https://www.flickr.com/photos/elaws/3773702375/sizes/l/