aureus-insights_logo

All Posts

Cheers to Stream-based Data Integration...and to Never Losing a Byte!!

Enterprise applications belong to a vibrant ecosystem and consequently the data they generate is large and varied. Enterprises both benefit and suffer from this nature of application and data.Whenever a new application is to be deployed in an enterprise that integrates with the applications in the ecosystem, the precondition is an 'expansive data definition with referential value' on day 1 to start integration. Traditionally, this approach to data integration involves identifying a target data structure, and force fitting data from all sources into it. This is done to ensure a 'seamless' integration - never mind the loss of data considered irrelevant.

Data Warehouse Approach

Traditional data warehouses use the following approach for data ingestion:

  • Fixed target structure in which to ingest data
  • Source data is transformed to fit into the target structure
  • Any 'alien' data is just ignored and dropped
  • Unstructured data is sparingly allowed
  • Reporting and analytics is run on top of this target structure
  • Structure is reviewed periodically for change in definitions.

So how do we absorb data without defining so that we can discover the structure and value of data on an ongoing basis?

Benefits of Stream-based Data Integration

In the real world, data gets collected or elicited through events - enterprise-initiated or customer-initiated. The data collection approach from transactional systems to analytical systems can follow a similar approach if event data can have a variable structure. This works best if data flows as streams from multiple sources. This approach is called event-based data modelling using data streams.

  1. Unit of integration is a data packet which contains a series of connected name-value pairs.
  2. Data packets flow into the data ingestion environment into one or more target streams. The packet is considered for processing on a stream if the minimum variables required for a stream are present in the packet.
  3. Each data packet can provide foundational information for multiple different events.
  4. Data is stored as events; data for the same event can be provided incrementally.
  5. The variables in a packet that are unknown to the data ingestion environment are not ignored or dropped Instead, they are retained at all events that are identified from the data packet.
  6. Since events are real world concepts, they form an excellent foundation for analytical models targeted towards behavioral outcomes.
  7. Data ingestion can start very quickly with conformance to a minimum set of variables required per stream.
  8. Adjunct variables can be discovered after ingestion and used for analytical value
  9. Works very well with the real-time paradigm in which the businesses of today compete.

In consumer-based enterprise businesses where relationships are long term and are influenced by experience, the following streams are essential:

  • Customer
  • Relationship
  • Transactions
  • Interactions

These streams abstract the natural structures of the processes that govern the operations of the business. And to top it all, it can be kick-started quickly and improved upon in an on-going basis.

Advantages of Stream-based Integration

The following are some of the technical advantages of stream-based integration:

  1. Cloud Technologies: applications like Salesforce hosted on the cloud can be connected to via Native API or integration APIs like MuleSoft by using adapters.
  2. Legacy application integration: legacy applications that allow connectivity via Message Queues or flat files can also integrate in a stream-based environment
  3. Batch-base upload: data which is available after EOD processing or is available from external systems as flat files can also integrate on streams
  4. Real-time integration: enterprises environments with ESD can easily connect in real-time to the web service end-point of the related stream. Also, applications that can call web services can also connect at real-time.

Conclusion

If you want to see analytics results in real-time, then stream-based processing is the way to go. A stream-based approach to data integration preserves the sanctity of data through its life cycle. Also, it fits in easily with many different environments or sources such as the cloud, legacy applications, batch uploads and real-time integration.


Download the Aureus Analytics whitepaper "Data Integration with CRUX" to learn how data stream-based integration is used to bring together multiple datasets in real-time.

Download

 

Nitin Purohit
Nitin Purohit
Nitin is CTO and co-founder at Aureus. With over 15 years of experience in leveraging technology to drive and achieve top-line and bottom-line numbers, Nitin has helped global organizations optimize value from their significant IT investments. Over the years, Nitin has been responsible for the creation of many product IPs. Prior to this role at Aureus, Nitin was the Global Practice Head for Application Services at Omnitech Infosolutions Ltd and was responsible for sales and profitability of offerings from application services across geographies.

Related Posts

3 Emerging Trends in Insurance, Social Media and Predictive Models

The use of social media is growing at a steady rate, and with that, the adoption of predictive analytics is on the rise as well.

How Intelligent RPA will Improve the Customer Experience

RPA is Changing Lives The digital marketplace is extremely challenging. Businesses must stay ahead of the game by being innovative and creative to continually provide automated processes that will improve the customer experience. Robotic Process Automation (RPA) and Digital Process Automation (DPA) are the front runners by being one of the fastest growing segments of business technology these days. Globally, the RPA market size was valued at USD 597.5 million in 2018 and is expected to register a compound annual growth rate of 31.1% between 2019 - 2025.

Understanding Your Customers - Now and in the Future

With the artificial intelligence (AI) technology that exists today, an AI-enabled single view of the customer provides insurers with an opportunity to improve the customer experience. In the previous blog article “The Importance of Having a Single View of Your Customer in 2019,” we discussed a few of ways this can be accomplished.