According to Ken Collier, “Agile BI” blends the iterative, quick progress of Agile Software Development and a short-run project approach to Business Intelligence using the same steps and measurable results. Collier is a senior consultant in business intelligence and agile product and project management for Cutter Consortium, an Arlington, MA, IT advisory firm of independent consultants.
Collier encourages many consulting clients to progress in baby-steps, producing bite-sized deliverables in rapid-fire, two-week cycles. The idea is to be constantly rolling out new features, offering a more nimble response to BI users’ needs and wasting less time on features that will never be used. In a “Smarter Technology” article he outlined why this adaptation is changing the game for both technology departments AND business operations.
Q: How is “Agile BI” helping companies move past the “platform” model of big BI projects and brand-specific installations?
A: Most recently I’ve been working in transport-logistics and agriculture sector and it’s largely been Agile and online/SaaS kinds of things. One of the challenges I bump into all the time is the constraints of what the IT team is comfortable with. Executives have big views of what they want to accomplish but there may be an artificial constraint for new, emerging BI technologies to get in because the IT team is more comfortable with specific packages such as Oracle or Cognos.
I’ve been a Spotfire fan for awhile but find it hard for IT shops to take the first bite. One of the bubbles that has been percolating in the last six months or so is Google Analytics and suddenly it seems like it’s on everyone’s desk if they’re doing web business and analytics. An e-commerce company had Google Analytics in their own world. Clients were using it and paying attention. Smaller companies are good prospects for SaaS analytics and that’s raising visibility for the need of analytics insights. Some companies are using internal enterprise warehouse for sales/leads data and HR – ERP. But companies are starting to get the need for analytics and Google or Salesforce are giving them some pretty powerful tools. Plus, the open source stacks are maturing.
Q: Cutter’s research on how the economic slowdown is affecting the BI market looked interesting. What insights can you share?
A: In the past year, companies went into hunker down mode. The last time they did that – in 2000-02 — they fell behind on BI. But in the past year or 18 months, I have seen companies doing iterative, creative, evolutionary things and spending a few hundred thousand dollars to take a step forward, instead of $12 million a huge new installation. One dealt with customer profitability and efficiency in managing their customers and funded at a moderate level. And my bent for Agile BI says these are more justifiable on smaller projects than multi-year, multi-million data warehouse projects. In the case of Agile BI my focus is on feature delivery and sometimes customers are doing strategic forecasting or customers in finance department or whoever. But Agile is about delivering high-value data.
Q: Do companies know what they need when they start inquiring about analytics or BI?
A: Companies may say ‘We’re the last to admit we don’t have an enterprise data warehouse.’ That’s the wrong-headed approach to pursuing a BI solution. And Salesforce or others like it are solutions to a business issue – where the needs are clear and specific. It’s surprising the number of IT professionals – not line of business people — who say what workers are going to use and identify their needs. Agile starts with a customer-need focus. And BI often has a multi-year timetable and companies say ‘Install it, then we’ll figure out how the business is going to use it.’ They let features drive the data mashup.
Q: One estimate from Cutter found only 15 percent confidence in the data most companies have – how is that affecting the BI market?
A: That’s something I deal with everywhere I go – the number seems low but is not really surprising. In a typical large enterprise there are lots of different systems and trying to pull all the data together is clunky. And inevitably there are problems and ‘dirty data’ but the traditional data warehouse is a lot of work to bring data together. They’re not able to understand the business domain experts — these are the people who can explain it, correct it, suppress it and find the right way to manage. If they’ve invested in a BI system, faulty data is being presented to end-users and they’re saying ‘Wait a minute – if this is wrong, how many other things are wrong? Ultimately the system has to evolve to get it where you want it to be – in a typical waterfall development it can be a long time before you get the feedback to clean the data or apply the business rules and incrementally then users feel more comfortable about the data itself. They do early buy-in.
Q: You point out that the flow of information is shifting from querying a database “PULL” to “Exception and pro-active PUSH.”. How is this changing the BI world with what you call message-driven architecture?
A: We built a system from external customer databases and SaaS where we had to deal with pushing data AT the system and I think there’s a lot of merit in that approach. It’s a bit of a tough sell when you try to get the IT shop to say we need to learn about message architectures and languages and object-oriented coding and XML format stuff. But that generalizable, adaptable architecture is possible to bootstrap any business domain and there may be some performance issues. BI systems are hobbled by the fact they are intractable the more feeders you have and more disparate and decentralized data becomes. I routinely hear people say we’d like to integrate data from one system or another – but we’ll have to put it on the wish-list. It can get costly to work with and extend outside the enterprise to your upstream partners, downstream customers. Those things become really great ideas but really costly because of the bandwidth issue.
Q: And what’s next in BI?
A: I see companies being limited by the knowledge inside their organizations. We’ve come a long way from where BI was 10 years ago but we’re still dealing with Excel spreadsheets and static reports that don’t let me do queries or analysis. I think data visualization is going to be the next big thing but I’ve been saying that for years. Companies have to crawl before they walk in the BI space and when they do crawl, they say ‘Okay that’s good enough.’ Rear-view-mirror modeling is sufficient for a lot of companies and they don’t want to take the next step in BI maturity.
