1Cloudhub (1CH) with its Framework-based Analytics Practice (BI & AI) has helped organizations to achieve their objectives. We have faced quite a number of risks and challenges in the process. We'd like to share our experience of addressing these challenges in this blog.
There has been an increasing need for Big Data Analysis in almost all business lines. Organizations of medium-sized to large enterprises, using different technologies, evaluate large data sets and draw conclusions about them that allow them to make informed business decisions.
We could classify the various risks/challenges encountered into 3 main categories :
Process Challenges
Customers most often have challenges in understanding their data and need support in aligning their business objectives to the extent that it helps them to identify KPIs. 1CH conducts a series of workshops with the Business Teams of the Customer. As a result of the workshops, we align their business objectives to ensure that KPI’s objectives are time-bound as a success criteria for engagement.
It is essential to understand the customer’s data security policy. Restrictions on access to production data during one-time ingestion and development of CDC ETL / ETL may exist often. Such policy restrictions are identified and mechanisms to address it, are discussed ahead of the project execution phase.
1CH thinks of Non-functional Requirements before the implementation process and offers them as part of the design, thereby eliminating complications and delays in actual project execution.
Technical Challenges
The most common challenge in any organization is to separate data sources. 1CH team pilots 1 or more Data Ingestion techniques and evaluates the best technique for customer data landscape using our robust Ingestion Framework, which can be configured for multiple data flows.
Many customers provide a huge ocean of raw data, with an exorbitant number of tables and fields, not all of which would be needed directly or indirectly to achieve the project objectives.
1CH uses data mapping logic and pulls just the required fields and builds an interim data view to help run queries more quickly, resulting in faster TAT for data fetching.
Disruption and instability of the network were major reasons for significantly delaying the time of data ingestion. We work closely with the Network teams of the customer to set up monitoring mechanisms to identify network disruptions and also set up preventive mechanisms to ensure stable network connectivity for seamless data flow from data sources.
Data Challenges
The lack of Database level CDC Strategy, is a common problem faced in number of customer engagements. There might be various reasons restricting customer to provide production data for only 1 ingestion period.
In such cases, we advise customers to determine a CDC strategy (hourly / daily data pull) that would suit their operational model and data security policy.
When QA environment data is used for the development of ETL, there may be challenges in missing cross table aggregation references because QA data is not of production quality. During the assessment phase, we conduct a data audit of the raw data shared by the customer, highlight these gaps and ask the customer for an updated version of the raw data.
Key data fields are sometimes prone to human error, including monitoring / language errors, and data is not consistent across data sources. We worked with the customer’s business team to develop a mapping logic for data cleansing to address these data issues and to re-share raw data for ingestion.
Learnings
1CH uses Nimbus Insights ® framework, which offers us the benefits below :
1CloudHub has helped multi-national enterprises move to the cloud without a glitch. We can help you. Speak to a cloud consultant today.
Updated on May 5, 2020
