Security Gateway Appliances

Companies report large investments in their data warehousing (DW) or Business Intelligence (BI) systems with a large portion of the software budget spent on extract, transformation and load (ETL) tools that provide ease of populating such warehouses, and the ability to manipulate data to map to the new schemas and data structures.

Companies do need DW or BI for analytics on sales, forecasting, stock management and so much more, and they certainly wouldn’t want to run the additional analytics workloads on top of already taxed core systems, so keeping them isolated is a good idea and a solid architectural decision. Considering, however, the extended infrastructure and support costs involved with managing a new ETL layer, it’s not just the building of them that demands effort. There’s the ongoing scheduling of jobs, on-call support (when jobs fail), the challenge of an ever-shrinking batch window when such jobs are able to run without affecting other production workloads, and other such considerations which make the initial warehouse implementation expense look like penny candy.

So not only do such systems come with extravagant cost, but to make matters worse, the vast majority of jobs run overnight. That means, in most cases, the best possible scenario is that you’re looking at yesterday’s data (not today’s). All your expensive reports and analytics are at least a day later than what you require. What happened to the promised near-realtime information you were expecting?
Contrast the typical BI/DW architecture to the better option of building out your analytics and report processing using realtime routing and transformation with tools such as IBM MQ, DataPower and Integration Bus. Most of the application stacks that process this data in realtime have all the related keys in memory –customer number or ids, order numbers, account details, etc. – and are using them to create or update the core systems. Why duplicate all of that again in your BI/DW ETL layer? If you do, you’re dependent on ETLs going into the core systems to find what happened during that period and extracting all that data again to put it somewhere else.

Alongside this, most organizations are already running application messaging and notifications between applications. lf you have all the data keys in memory, use a DW object, method, function or macro to drop the data as an application message into your messaging layer. The message can then be routed to your DW or BI environment for Transformation and Loading there, no Extraction needed, and you can get rid of your ETL tools.

Simplify your environments and lower the cost of operation. If you have multiple DW or BI environments, use the Pub/Sub capabilities of IBM MQ to distribute the message. You may be exchanging a nominal increase in CPU for eliminating problems, headaches and costs to your DW or BI.

Rethinking your strategy in terms of EAI while removing the whole process and overhead of ETL may indeed bring your whole business analytics to the near-realtime reporting and analytics you expected. Consider that your strategic payoff. Best regards in your architecture endeavors!
Image by Mark Morgan.

Wouldn’t it make more sense to deal directly with IBM?

This is a topic of much discussion around the TxMQ office these days, as it is at other solutions providers. Prospective customers often ask us: If my company’s able to go direct with my manufacturer/vendor (be it IBM, Microsoft, Oracle, or whomever), why shouldn’t we?
It’s a fair question. Companies, especially large ones, oftentimes have multilayered supply-chain relationships for acquiring software, hardware, and talent.
On the one hand, long-standing, embedded relationships often dictate the way a company acquires technical solutions. A friendship nurtured through years of trusted business dealings – sometimes called a “trusted advisor” relationship – may be the perfectly legitimate and ideal way of solving technical challenges. Yet sometimes things change. What happens when your salesperson – the same salesperson who’s covered your company for years – resigns?
Back to the broader topic of direct or business partner: As software and hardware companies like IBM and the other majors evolve over time, their business models push more sales through channel partners. Business partners are inevitably smaller companies, and are typically far better equipped to build and maintain longstanding, deep customer relationships. Sales teams at the major vendors change, oftentimes annually, leading to spotty coverage of accounts, and occasionally even leaving some companies with no direct coverage at all.
Business partners typically offer more consistency and stability of coverage. In addition, as IBM and the majors continue to add layers of complexity to their brands, and shift products in their portfolios, it’s very difficult for companies (and even their field salespeople), to keep products straight.
Surprisingly, this is a knowledge area in which partners tend to excel. Most of the leadership at business partners were themselves former employees of the majors. They left the majors to explore greater freedom to engage with customers, and to build deeper, better relationships over longer periods of time without the bureaucracy that comes with working in a large shop, or the threat of annual account “realignment.”
Also, as solutions offered to customers become more complex and cross over traditional brand borders, business partners are better able to navigate these tricky waters.
Recently IBM, along with other majors, went through internal realignments that left salespeople covering products new to them, and others were shifted to entirely new lines of business. Not so with business partners, who are free to engage as they always have.
What about software and hardware sales? Logic says it must be cheaper to buy direct. No indeed. IBM, as one example, has dramatically shifted internal teams, and reduced field and inside sales coverage to better align their resources with today’s market. What does this mean? It means it’s usually cheaper to order software and hardware from a partner. IBM knows it’s very expensive to have a large, geographically dispersed sales team, and far more cost-effective to let IBM business partners sell more and more software, services and hardware for them.
Partners therefore have full access to special bid requests, discounting, plus any and all sales tools a direct seller has in the arsenal. In addition, it goes without saying that a business partner’s services rates are nearly always below IBM direct rates.
Conclusion
In the end, each company must decide what’s best for itself, but don’t presume that the way you engaged with IBM and others in the past is the best way to engage in the future. A business partner can be your best ally to stay current with technology, and enable the nimble, robust infrastructure your company needs to compete and win in today’s marketplace.
Let’s take this conversation a bit further – email me at [email protected].
(Image by Flazingo Photos)

Tag: Security Gateway Appliances

Why ETL Tools Might Hobble Your Real-Time Reporting & Analytics

Why Should I Use An IBM Business Partner?