Feeds:
Posts
Comments

Posts Tagged ‘datawarehouse’


Big data requires extreme workloads

Read Using Big Data for Smarter Decision Making by Colin White.

Big data involves more than just the ability to handle large volumes of data. It also represents a wide range of new analytical technologies that opens up new business possibilities. But before reaping the rewards of big data analytics, there comes a set of challenges around deploying new technologies into existing data warehouse environments and providing systems that optimize computing performance for different workloads.

As I explored in my recent posts on smart consolidation, the data warehousing and analytics environment is more complex today than even just a few years ago. Many have found that mixing operational analytics and deep or advanced analytics on the same system brings significant challenges to performance and meeting SLAs. With operational analytics, business managers need continuous data ingest and fast access to standard reports with the ability to perform ad hoc queries that drill down into the data and provide new perspectives and insight. When a deep analytical query comes along that requires significant data volumes and extreme computing resources, operational query performance suffers. Big data adds yet another complexity around data sources, data quality, longevity of the data, and whether some of the big data should be integrated into the enterprise data warehouse for longer-term historical analysis.

The best way to handle these different types of workloads is to optimize systems to the workload, and combine these solutions with the enterprise data warehouse to create an “analytical environment”. We see many types of optimized systems in the market today – data warehouse appliances, data marts, noSQL systems, Hadoop-based systems, streaming data analytical systems, cloud-based solutions, etc., that complement (not replace) the enterprise data warehouse. Each system is optimized for a specific workload, and used together they can help streamline and provide fast response to a wide variety of business needs.

A majority of organizations today already understand this – really, optimizing computing resources to various types of data and associated workloads is nothing new. At some point in the data warehouse and analytical environment evolution, organizations reach a tipping point that drives separation of data and workloads. Data growth and new sources of (traditional) data, an increased number of users, increased complexity of queries, and “big data” are all drivers of this tipping point.

Colin White of BI Research wrote a white paper exploring new developments in data warehousing and analytics and the benefits that analyzing big data brings to the business. The paper also reinforces this notion of optimizing systems based on the types of data and workloads. The conclusion – integrating these systems together into a single analytics infrastructure drives smarter and faster business decisions. Read Using Big Data for Smarter Decision Making.

Advertisements

Read Full Post »


A modern data warehousing and analytics architecture

Consider the example of a credit card company.When a customer applies for a credit card, the sales department collects the customer’s details and financial history, and the compares it to historical data from third-party reporting agencies to determine the customer’s ability to manage and repay debts. The customer data flows to the marketing department, where it is analyzed for trends and compared with opinion content collected from the Internet to make decisions on promotional campaigns.

Eventually, the customer might request a credit-line increase, at which time the customer service system will recommend up-sell opportunities and the lending department uses the customer’s payment history to evaluate the request. Meanwhile, the company’s online transaction processing (OLTP) systems are fielding millions of transaction authorization requests per minute. Real-time analytics systems are looking for anomalies that may indicate fraud by comparing the streams of transaction data to patterns developed by analyzing customers’ purchasing histories. As all this data ages and becomes more static, it shifts to archival systems and is stored using specialized technologies like Apache Hadoop—yet it remains available for instant auditing and long-term trend analysis.

At the same time, the marketing department is investigating a new customer segmentation model to use in an upcoming product launch. Marketing has been busy analyzing their complete customer database to determine online banking trends as well as smart phone and mobile banking adoption rates. After many iterations of their segmentation model, they believe they have identified the data elements and customer behaviors that define a financially sophisticated and technologically savvy customer segment. Now, several months prior to the launch, the product manager is running predictive models to test the business case on combinations of marketing messages and user adoption rates. The team is free to test and retest their assumptions, even though their queries take a long time to execute, because they are running on an analytics-optimized data warehouse appliance—not the primary operational analytics system.

The credit card company is taking advantage of distributed data and a distributed workload architecture. By intelligently separating workloads, it is able to creatively analyze data to identify new business models, test assumptions for new paid services and optimize launch and execution plans without impacting the daily, hourly and up-to-the-minute operational needs of its core business.

Read Full Post »


Many companies have found success in building data warehouses that meet basic needs, but are now finding they need to move beyond the back-office warehouse to leverage information on the front lines of decision making throughout their entire company. They need information on demand and need the ability to build systems that can deliver on those promises with real incremental returns.

For those who understand the power of an analytics-driven organization, this is a most exciting time. The opportunities are limitless: customers, prospects, suppliers and the business itself are creating endless geysers of data. Analytics tools are inexpensive, widely available and so easy to use that they make business sense in almost any situation.

To move forward, organizations need a strategy that delivers on several focused business requirements:
1) Operational management: Accelerate time-to-market to meet business SLAs for new and existing business processes, operational analytics and business intelligence (BI).
2) Big data: Leverage unstructured data, social media and other “big data” information sources to gain more insights from more data—without impacting the business SLAs.
3) Predictive analytics: Forecast future trends and analyze risks and potential outcomes.

Many IT organizations are adopting a strategy called smart consolidation that reconciles the need to simultaneously distribute data warehousing and analytics capabilities and infrastructure while centralizing management. Smart consolidation is a method for evolving an existing data warehouse architecture to meet today’s demanding analytic needs, such as big data, streaming data and unstructured data.

In a nutshell, it involves thinking beyond the traditional warehouse structures that have provided great success with structured data, basic reporting and analysis. Smart consolidation is driven by these four goals:

  1. Consolidate and govern enterprise data
  2. Optimize workloads for performance and SLAs
  3. Simplify the delivery of analytics by leveraging appliances
  4. Flexibly extend analytic capability as needed

The basis for smart consolidation is to completely optimize an analytics architecture by placing the right workload against the right data, in the right place, at the right cost and the right performance level.

Smart consolidation acknowledges that an organization requires different types of databases, analysis tools and data formats. It needs traditional data warehouses, data warehouse appliances and operational BI systems that can accommodate different types of workloads. It also needs systems based on advanced technologies that can efficiently handle data that is moving extremely quickly as well as large volumes of data that does not change frequently.

Single system? I think not

No single, data system could efficiently serve all these requirements and perform well for both transactional and analytical workloads. Under the smart consolidation strategy, multiple specialized elements use industry standards to communicate and join together to form a fluid, agile data ecosystem that delivers business insight, cross-organizational data governance and centralized IT resource management. By allowing many different elements to serve specialized needs, smart consolidation also enables organizations to accommodate the endless variety and rapidly growing ocean of semi-structured and unstructured data.

Read Full Post »


Find out more: http://www-01.ibm.com/software/data/infosphere/warehouse/
Find out more: http://www-01.ibm.com/software/data/db2-warehouse-10/

Scenario: A retail store is now able to predict and replenish its merchandize stock by operationalizing an automated process flow. The retailer can analyze inventory data and product stock levels in real time. By deploying IBM InfoSphere Warehouse to reduce decision latency, the store’s accelerated decision-making capability now effortlessly keeps pace with the speed of demand.

Key benefits of InfoSphere Warehouse v10:
Faster Insights

  • Real-time Business Intelligence as operational data is continuously being feed into the warehouse.
  • Faster, accurate decision making, turnaround times.

Improved Cost Efficiencies

  • Adaptive Data Compression provides on average 30% improvement (up to 75%) over IBM’s existing Deep Compression.
  • Multi-Temperature Storage allows you to optimize data storage cost-efficiency.

Higher Performance

  • Star Schema optimization delivery for quicker response times – delivering 3x performance on BI workloads.
  • Continuous Ingest of data optimizes loading of data leading to faster decision making.
  • High Availability Operational access concurrent with analytics.

Increased Team Productivity

  • Built-In Time Travel query enabling faster historical and trend analytical queries.
  • Row and Column access controls to support multiple tenant operational warehouses.
  • Native Bi-Temporal support improves developer and DBA productivity.

Read Full Post »


Good data is attained by integrating multiple data sources, deriving a ‘single version of the truth,’ and putting that good data (and unstructured content) into a data warehouse where the BA/BI tools can perform their magic. DDD (data-driven decision making) begins and ends with good data.

“smarter technology” posted a great article “Data Two, Gut One” that got me thinking more about the value of good data. In the article, they state that new research shows data-driven decisions improve organizational performance and company value.

In my current position as a marketing evangelist for data warehousing and analytics, that has proven itself time and time again. When looking for ways to increase our marketing effectiveness, I look at the various marketing channels we use to get our messages out to the market. Having complete data available regarding marketing activities, marketing channels used, customer segmentation sets, response rates…and, how leads progress through the sales funnel, is critical.

When some of the data is unavailable, quite simply, the “truth” is not known, and any decisions based on this information is just a guess, gut feel, intuition, a hope and a prayer, a SWAG. When the data is complete, accurate and trusted, I can then make quality decisions to fill in any gaps, go after new markets, tweak the messaging to get higher response rates, etc.

Analytics applications that nicely present dashboards, scorecards, historical trends, predictive analys, and give me actionable insights, can all benefit from good data. Good data begins with data integration, data quality, and a good data warehouse.

If anyone reading this post has had good experience with good data, or a bad experience with bad data, I encourage you to share your story by commenting.

Read Full Post »


Pay only for the data you analyze with InfoSphere Warehouse 9.7.3
Download this whitepaper

The new IBM InfoSphere Warehouse 9.7.3 offers a simple and transparent “pay for what you use” terabyte pricing that allows your organization to pay only for the volume of compressed user data you manage in your data warehouse. Download this whitepaper to learn how to build high-speed, scalable analytics into your data warehouse with InfoSphere Warehouse.

As your data volumes grow, you can purchase additional warehousing capacity on a just-in-time basis. IBM’s industry leading compression capabilities further increase the value of this new pricing option. Do what’s right for your business and your users – put the best warehouse software on the best machine for the job.

Now pay only for the data you analyze with the new InfoSphere Warehouse 9.7.3 and deliver powerful business insights. Download this whitepaper to learn how to build high-speed, scalable analytics into your data warehouse with InfoSphere Warehouse.

Read Full Post »


Great article on Business Week site… http://www.businessweek.com/managing/content/feb2011/ca2011023_063316.htm?campaign_id=yhoo 

Uncertainty… To mitigate uncertainty you need insight into the marketplace, your business, customer behavior, economics, supply chain, etc etc etc. Where do you get that insight?? From the data sources within your company and from data outside your company. How do you analyze it?? With good data warehousing solutions, good data models and business intelligence / business analytics tools. Will this get you ALL the answers you need?? NO, but it’s far better to do your analysis and make as many decisions as you can based on facts, analytics, trends, scorecards, dashboards, performance, predictive modeling, than it is to rely on gut feel, intuition, or that arrogant word, experience. Data warehousing and analytics are seeing a huge surge for this very reason – to take as much uncertainty out of the equation as possible and base decisions on facts and analytics rather than gut feel. See more here… http://www-01.ibm.com/software/data/infosphere/data-warehousing/.

Read Full Post »

%d bloggers like this: