Quantitative Analytics: Data Growth Drives New Storage Demands

Today’s quantitative finance environment is dramatically different from that of just a few years ago. We’ve gone from gigabytes of daily market data that is put in memory for algorithms and jobs isolated to a few machines, to terabytes of data and systems that need to process hundreds of thousands – and even millions – of jobs.

Read More: E20-562: VPLEX Specialist Exam for Systems Administrator (DECS-SA)

As financial firms seek faster, reliable and more effective order execution that can also help reduce transaction costs, algorithmic trading has experienced tremendous growth. With an expected CAGR (Compound Annual Growth Rate) near 12% through 2026, algorithmic trading will continue to fuel a surge in data and the adoption of data-intensive technologies designed to optimize insights and maximize performance, such as artificial intelligence (AI) and deep learning.

This exponential growth in daily transactions means that algorithmic quant trading firms can no longer store the active trade data sets in memory. Instead, they need to transition to support growing quant teams on increasingly larger data sets – at scale. This creates new performance requirements compared to what was once fairly simple back in the days when infrastructures were very fast, and data could fit in RAM. But the explosion of data and the need to collect, store and analyze more data than ever, has changed – and complicated – environments. Terabytes of data are now generated by smart systems that are more powerful, can handle billions of market transactions, and workloads that perform risk modeling, portfolio management, simulations and other data-driven computations.

From a storage perspective, often SAN and NAS haven’t been able to scale to the volume of data that is being ingested today and, as a result, performance suffers. Without advanced storage, firms would need to reduce the data sets they are working with, give up near real-time performance, or limit the number of simultaneous processes. In today’s competitive market, that’s not a viable option.

Effective quantitative analytics requires the following storage features:

◉ Highly scalable storage to efficiently store billions of files

◉ High read/write performance

◉ High user concurrency capability

◉ Fast interconnect between compute resources and the storage platform

What’s needed? Storage that supports extreme concurrency at scale

High frequency trading and analytics environments are characterized by massive data volumes, low latency and complex processing capabilities. So, it’s not surprising that infrastructure and data modernization have become a priority for quant trading firms. Organizations are moving from infrastructure that supports real time, small data sets to large, highly concurrent workloads with large data sets. Organizations also need to be able to securely support data growth for concurrent workloads to comply with governmental and regulatory standards, even as the number of people simultaneously accessing data grows as well. With financial services experiencing a 1300%+ increase in ransomware attacks in 2021, data security and risk mitigation need to be foundational in today’s quantitative analytics environment, both financially and operationally.

The ideal storage solution enables firms to:

◉ Leverage single copy architecture with a centralized shared storage model. This makes managing a single copy of all market data easier than managing a copy for every compute node.

◉ Run analysis and AI/DL workloads simultaneously with co-located data that supports both types of jobs and compute. This eliminates data movement or specialized configurations.

◉ Leverage massive scalability and simplicity by storing data under a single file system and namespace for all applications to use. This provides less operational overhead while allowing you to non-disruptively scale capacity.

◉ Mitigates risk by increasing resiliency and minimizing vulnerability to cybersecurity threats.

Dell Unstructured Data Solutions for quantitative analytics – Storage that performs, scales and secures

With Dell Unstructured Data Storage (UDS) solutions, financial firms can leverage a portfolio that meets the performance, scalability, and security demands of quantitative analytics environments. In addition to solutions with scale out architecture to enable high bandwidth, high concurrency, and high performance with all flash options, UDS is uniquely suited for enterprise-grade trading:

◉ Leverages efficient single copy architecture for market data

◉ Runs tick data analytics and AI/DL workloads simultaneously with data in place

◉ Runs in multi-cloud with data sovereignty

◉ Supports a range of core trading technologies including Kdp+, a leader in high-performance time-series database systems from KX Systems

The Dell OneFS operating system does not rely on hardware as a critical part of the storage architecture. Instead, OneFS combines the three functions of traditional storage architectures—file system, volume manager, and data protection—into one unified software layer to create a single, intelligent file system that spans all nodes within a storage system. Unlike traditional storage systems that have a finite maximum size and must be replaced by a bigger storage array when the maximum performance or capacity is reached, a OneFS powered cluster can linearly expand, or scale out, performance and capacity to seamlessly increase the existing file system or volume into petabytes of capacity.

In addition to built-in availability, redundancy, security, data protection and replication with OneFS, the UDS portfolio provides protection from cyberattacks with integrated ransomware defense and smart AirGap. Superna Ransomware Defender detects suspicious behavioral patterns across multiple vectors, report on operations and event data in real-time, and block active threats.

The benefits for quantitative finance firms?

◉ Shortened model development time

◉ Faster analysis on larger, multi-day trading data

◉ Extreme performance for smaller real-time data

◉ Accelerated cycles of learning by bridging historical and real-time databases

◉ Ability to conform to regulatory standards and guarantee enterprise data protection

Discover how Dell Unstructured Data Solutions helps customers modernize quantitative analytics infrastructure to deliver high performance and extreme concurrency at scale. Contact your local Channel or UDS manager for additional information.

Source: dell.com

Continue reading

Driving the Next-Generation Sales Solution on SAP HANA

As EMC IT remains focused on its IT Transformation journey, we continue to take calculated risks in emerging technologies and adopt bleeding-edge solutions — but not without bumps along the way. However, we have found that leveraging a strong partnership with a trusted supplier is a crucial strategy to help smooth the road ahead.

Recently, EMC IT went live with the third release of its SAP implementation, replacing our direct and channel ‘selling’ tools with a suite of software products predominantly within the SAP landscape. This grouping of products – while not currently sold by SAP as a “solution” – needed strong partnership between EMC leadership and SAP, as we cooperated to string them together.

Back in 2003, EMC’s ‘selling’ tools, Direct and Channel Xpress, went live as two separate applications as a conscious choice in a very different selling model. At the time, EMC’s direct sales force was focused on high-end deals with little to no partner interaction.  But as the channel grew and the go-to-market model adapted it became clear that this (EMC+Channel) partnership needed a more tightly coupled application – one that we just couldn’t deliver with two disparate applications.

Choosing the New Solution

The core of the new application is SAP’s CRM (customer relationship management). EMC IT quickly made the decision to run it on SAP HANA database platform – our first venture  to deploy an on-line transaction processing (OLTP) platform using SAP HANA. The capabilities within SAP CRM for quoting, pricing and customer management were going to be the core of the entire system.

The next major decision focused around the next-generation product configurator. The unlimited configurability of our products and solutions is a major competitive advantage for our sales force, and EMC needed an agile tool to maintain these tens-of-thousands of rules on a daily basis.  We moved forward with a plan to use SAP’s Solution Sales Configurator (SSC) – a relatively new product with only one other production customer at the time.

It was quickly apparent that SSC’s ability to be modular and configure solutions rather than individual products matches EMC’s vision as our product portfolio continues to grow. But like all new products, there are growing pains as the application evolves. SAP stepped up to the task and directly partnered with EMC  – from senior leadership level and down – prioritizing bug fixes directly affecting EMC and enhancing the tool as we found deficiencies to our process.

The last product selected was Vendavo – a leader in the Pricebook and Deal Management space as well as a close partner of SAP.  Touting out of the box integrations, Vendavo was a natural fit for us.

The “Wow” Factor

With the main components selected, the focus quickly turned to user experience. Ensuring ease in selling EMC’s products would make the salesforce more efficient as well as help our partners choose to sell EMC instead of one of our competitors.

Our first user-experience step was to enlist the help of SAP’s Custom Development organization to help build a tool to easily create standardized UI’s on top of our complex product configurator. Using SAPUI5 technology and partnering with the SSC product team, SAP delivered a robust tool that allows the business to use drag and drop technology to create easy to use UI’s so sales engineers can quickly configure products.

As the project continued, in pure EMC IT fashion, along came a new curveball. SAP had recently purchased an eCommerce market leader – a software platform known as Hybris. EMC had used Hybris sparingly, most notable in its online eStore, but this was a large-scope change very late in the project timeline. Again, partnering with SAP allowed us to get our hands on early releases of its integration roadmap using SAP’s RFC jCo connections. A stateful connection (one where the client stores and re-uses the backend session) between Hybris and the SAP CRM backend allowed us to use Hybris as simply the front-end window into the quote while still taking advantage of Hybris’ core capabilities – especially its versatile product catalog, which EMC uses extensively for search and enrichment of products. EMC IT then reached out to Pivotal Labs – an EMC federation company who had done numerous user experience projects. Their design services approach, a skill that most IT shops simply don’t have, led to major improvements and efficiencies in laying out the ‘price quoting’ user experience.

Stability and Performance

Tightly coupling these software products and ensuring consistency between each is the main challenge for us as we look to stabilize the solution. Our UI strategy strives to make each disparate system feel seamless so users don’t realize they are switching to a completely different tool each time. This raises the need for data consistency between all applications – real time. Using a mix of direct web services and our Application Integration Cloud (AIC) technology (a custom EAI layer built on Pivotal SpringSource) for real time integration, the data flow is constant each time a price quotation is saved (which is done often to avoid data loss).

SAP has partnered with us again, sending out their brightest minds from Germany to optimize the solution. With a mix of short-, medium- and long-term changes planned in both the configuration and pricing area, the goal is to not only improve the end-to-end experience but optimize business processes and the user experience.  A range of changes are slated for Q1 (improving the run times and handling quote saves more efficiently) with much larger enhancements in mind for the coming quarters and beyond.

Continue reading