Recently I was in a discussion where the capacity for OEID to rapidly ingest data was highlighted as one of its highest value items. That’s a fair point. In the spirit of “Agile BI” the turnaround time for ingesting a brand new data source is indeed a key differentiator.
The risk is that rapid ingest in the spirit of “Agile BI” runs the same risks as “Agile” projects that cut corners and throw out discipline. Agile I believe means engaging with clients, embracing change, and building a solution through iterative improvements.
Rushing anything will not produce a quality product. Rapid data ingestion is a wonderful ability, but always ask yourself these questions:
#1- Do I understand the overall vision?
Is the data I’m loading going to be nothing more than a temporary throwaway data store? Or will it be there for the long haul, adding depth to existing records and needing to align to the existing dimensions and records?
There’s no harm in spinning up a data store and throwing it away after some quick exploration. That’s how experimentation and learning work. However making that the standard practice instead of supporting a more holistic enterprise architecture may not deliver the value your client needs. Consider how your data set might fit into the whole, it’s rarely without connections that need to be understood.
#2 – Am I delivering what the client needs?
Oracle Endeca Server will support you loading data without any structure at all. You can baseline a data store and attribute names and types can be reset anytime. Very accommodatingly all the data can default to strings and Studio is wonderfully intuitive for text searching.
This doesn’t mean you’ll answer all the questions your client will have. Ensuring numerical and date/time attributes are in their respective formats may ensure trending and statistical questions can be answered. Taking some time to consider the semantics and even attribute groups may make the data exploration a much richer experience for the client. Before you start madly shoveling data into a heap take pause to confirm it works for your client. The technology doesn’t require it, but there is return value from the data modeling exercise.
#3 – Am I developing like a cowboy?
This starts innocently enough, with a question being answered with a shrug and the suggestion it doesn’t matter. Maybe it doesn’t, but do this too many times and your solution won’t matter either.
There are good reasons for standards and best practices. Ignoring them for the sake of fast data ingestion can translate into such poor reporting quality that users lose confidence and trust in the system. Developers need to stay disciplined so their solution will also be measured by its reliability.
So yes, the technology in OEID enables very rapid ingest of data. Agile BI is very compelling that way.
Just never forget that quality products come from quality effort, and speed is not the only measure of success.