Source System Changes made easy
In real world environments changing source systems can easily break the whole Business Intelligence System.
Using native Database Tools like Microsoft SQL Server Integration Services give you a lot of flexibility to build the Data Integration Solution you need. Just identifying where you used a field that no longer exists can be a huge pain and literally take days in complex systems.
How Halo Source handles Source System Changes
Halo Sources stores Information about the Source Metadata in each Project. This way you know exactly which fields are used in which ETL Package or transformaton.
To check if your Data Integration is in Sync with the Source System you can simply Refresh the Source Metadata.
All Items that are no longer available or need your attention are color coded with a red dot.
You can open the Items and choose how to fix the Issue. You can use Default values, SQL Expressions, link to other fields.
With this Technology identifying issues from changed Source Systems becomes very easy and straightforward.
If you do not use Halo Source you have to identify the changed or missing fields manually.
To achieve this you need to
- manually gather information about what has changed
- open all SSIS Packages one by one in Microsoft BI Develoment Studio
- Check every Dataflow if you are using a Table that has changed
- if a table has changed you need to update the strucutre manually in your Staging area
- then you need to go through all steps of the data flow and add/change the field in the data flow
- then you need to double check if the change broke another package
Real world projects often consists of several some 50 Integration Services packages loading several hundred tables. To work through this project and make the proper changes can take several weeks of work. Especially if the documentation is not up to date.
In a Source Project with some 50 ETL Packages and several hundrred tables you can identify the whole impact of all source System changes in a few minutes.
All you need to do is to refresh metadata and ALL required changes are marked with a red dot.
The effort to fix these depends on how you get the correct values. In most cases it takes less than 5 minutes to fix a single issue.
Data Discovery
Source Data is sometimes hard to understand. Halo Source makes this process easy with its Data Discovery Features.
Here is an overview on some of these Features:
In a real wold System you often have literally hundreds of Fields with more or less meaningless names. The integrated Data Profiling allows you to sneak peek into your data and shows you for every field
- if it is populated
- the percentage of empy values
- how many Differnt Values you have
- The Top 5 ,Bottom 5 Values and the most common 5 Values
this makes it much easier to identify the right column. You literally see the values you are looking for immediately.
To get a deeper understanding of your Data you can see Data distribution (how often every value occurs). Gaps and Island Analysis is also available to show you if you have gaps in your Invoice Numbers.
To make sure your data looks as you expect it you can do a Pattern analysis showing you how digits, characters and Special characters are distributed.
You can store all profiling results in a database and monitor changes over time. This way you can easily identify unexpected Values and other Data Issues. Data Quality can be monitored over time. Halo Source keeps track about every field you used in the whole Data Transformation process. In the Global Table / Field Search you can identify every reference to a specific Field.
Here you see the global search showing all references to a Customer Code. No matter if it’s used in the Extract or a transformation – all of them are immediately accessible.
This makes it easy to see if you already extracted the needed information from the source system. If this information is available in your Source System, Halo Source can identidy how to link tables and allow you to add referenced Tables to your Project.
Data Integration:Saving 90%-95% Time and Effort
Save 90%-95% time and Effort in Data Integration
We developed Halo Source to speed up development of Data Integration Solutions. We make maintenance as simple as possible.
In a Real World Projects a lot can go wrong. Unexpected source system changes can cause weeks of work fixing the Data Integration. Employees who know all the details leave the company and nobody knows how to maintain their projects. Often there is no (or outdated) documentation. Requirements are often specified only on a very high level.
Halo Source addresses all of these issues. It reduces the the complexity of Data Integration Projects. Halo Source is accessible for non experts users. It empowers your team to work with agile methods.
This literally saves 90% time and effort and more compared to native Database Tools.
Here are some Areas where Halo Source helps a lot:
- Difficult Data discovery
- Source System changes
- People who designed Data Integration left the Company
- Special Skills needed to Maintain Data Integration
- Lack of Documentation
- Limited Expert Availability
- Changing Requirements
- Incomplete Requirements
