The ETL (Extract-Transform-Load) are one of the most critical processes of BI and Data Warehouse applications.

What is ETL?

The Extract-Transform and Load process consists of the following three sub-processes which are used to transfer data from production systems to the data warehouse where they are consumed by BI applications:

         •  Extraction of the data from production application and databases.
         •  Transform the data to reconcile it across source systems, including required data cleaning. Also the             data is transformed to meet the requirements of the target systems(StarSchema,Slowly Changing             Dimensions,etc).
         •  Loading of the transformed data into the Data Warehouse, Data Marts and other BI applications.

Enterprise ETL Vendors:

   DataStage and Informatica are examples of Enterprise ETL solutions that have extensive background and experience in the ETL market with their ability to scale performance in handling very large data volumes in complex, heterogeneous environments. These products provide comprehensive features and functionality and so require extensive training to use effectively.

How Competent Systems helps:

Competent Systems goal is to help customers optimize the value of all their information assets. We deliver the high-quality, low-risk information management solutions needed to boost business results. Competemt System's solutions are designed to,

         •  Reduce costs.
         •  Increase sales revenues and shareholder value.
         •  Monitor pre-implementation ROI projections with post implementation ROI results ensuring that our             solutions deliver as promised.

Informatica:

   The Informatica solution for enterprise data warehousing is proven to help IT departments implement data marts and departmental data warehouses and readily scale them up to enterprise data warehousing environments. This solution serves as the foundation for all data warehousing and enterprise data warehousing projects. It accelerates their deployment, minimizing costs and risks, by ensuring that enterprise data warehouses are populated and maintained with trustworthy, actionable, and authoritative data.

A Proven Solution for Enterprise Data Warehousing:

The Informatica solution for enterprise data warehousing empowers the IT team to easily adapt to changing business requirements. With the Informatica solution for enterprise data warehousing, your IT organization can provide your business with the data it needs for:

         •  Deeper competitive insights.
         •  Faster decision-making abilities.
         •  More comprehensive visibility across business units.
         •  Greater transparency for regulatory compliance.

ETL (Extract, Transfer and Load) Accelerator:

The ETL architecture will give BI and other projects a jump start in analyzing, designing and building ETL interfaces. Informatica provides specific insight for common ETL programming patterns such as common error handling mappings and application configuration guidance. The designs accompanying the code samples should be considered vendor-independent and can be leveraged in ETL projects, whether hand-coded or built from other vendor tool suites.

End-to-End Data Migration:

Most projects design and build conversion architectures using customized code, which increases the costs for Competent System's and its clients because the development process is slow and requires a high number of skilled resources. Competent System's Solution Works provides end-to-end conversion services by utilizing Informatica's ETL software to perform conversion development instead of building customized code. This conversion service offering saves at least 30 percent in conversion costs and reduces conversion development time by several months.

Datastage:

   DataStage integrates data on demand across many systems via a high performance parallel framework, extended metadata management, and enterprise connectivity.

   It integrates data on demand with a high performance parallel framework, extended metadata management, and enterprise connectivity.

         •  Supports the collection, integration and transformation of large volumes of data, with data structures             ranging from simple to highly complex.
         •  New features, such as an operations console, an interactive debugger for parallel jobs, and support for             balanced optimization, help customers work smarter, enhance productivity and accelerate problem             resolution.
         •  Offers a scalable platform that enables companies to solve large-scale business problems through             high-performance processing of massive data volumes.
         •  Supports real-time data integration.
         •  Enables developers to maximize speed, flexibility and effectiveness in building, deploying, updating and             managing their data integration infrastructure.
         •  Completes connectivity between any data source and any application.

Features and Benefits:

The powerful ETL solution supports the collection, integration and transformation of large volumes of data, with data structures ranging from simple to highly complex. IBM DataStage manages data arriving in real-time as well as data received on a periodic or scheduled basis.

The scalable platform enables companies to solve large-scale business problems through high-performance processing of massive data volumes. By leveraging the parallel processing capabilities of multiprocessor hardware platforms, IBM InfoSphere DataStage Enterprise Edition can scale to satisfy the demands of ever-growing data volumes, stringent real-time requirements, and ever shrinking batch windows.

Comprehensive source and target support for a virtually unlimited number of heterogeneous data sources and targets in a single job includes text files; complex data structures in XML; ERP systems such as SAP and PeopleSoft; almost any database (including partitioned databases); web services; and business intelligence tools like SAS.

Real-time data integration support operates in real-time. It captures messages from Message Oriented Middleware (MOM) queues using JMS or WebSphere MQ adapters to seamlessly combine data into conforming operational and historical analysis perspectives. IBM InfoSphere Information Services Director provides a service-oriented architecture (SOA) for publishing data integration logic as shared services that can be reused across the enterprise. These services are capable of simultaneously supporting high-speed, high reliability requirements of transactional processing and the high volume bulk data requirements of batch processing.

Advanced maintenance and development enables developers to maximize speed, flexibility and effectiveness in building, deploying, updating and managing their data integration infrastructure. Full data integration reduces the development and maintenance cycle for data integration projects by simplifying administration and maximizing development resources.

Complete connectivity between any data source and any application ensures that the most relevant, complete and accurate data is integrated and used by the most popular enterprise application software brands, including SAP, Siebel, Oracle, and PeopleSoft.