From hardware and infrastructure investments to data capture and storage issues, managing tick history data can be a challenging and costly undertaking. This blog highlights these complexities faced by financial services firms, and explains how LSEG Tick History – PCAP stored in the cloud with AWS S3 Direct provides an efficient and cost-effective solution that reduces TCO while still offering the level of data detail and quality that firms need.
- Capturing, storing and managing accurate and comprehensive tick history data can be both challenging and costly.
- Financial services firms need to consider numerous factors when it comes to managing their tick history data.
- LSEG Tick History – PCAP stored in the cloud with AWS S3 Direct provides an efficient and cost-effective solution that reduces TCO while still offering the level of data detail and quality that firms need.
To gain a competitive advantage in electronic trading, financial institutions develop complex algorithms to analyse market microstructure, identify trading patterns, and formulate effective trading strategies. To achieve these goals, access to high quality historical market data is essential. Tick data must exhibit exceptional levels of accuracy, accessibility, completeness, and usability, all captured using the lowest possible latency network architecture. With market participants continuously fueling the “arms race” to electronic trading – and the increasing quantification and granularity of timestamps from milliseconds to nanoseconds – access to the highest quality tick data has become imperative.
However, capturing and managing market data can be a costly challenge. Maintaining an in-house technology solution to source and capture market data takes time and is resource-intensive. Financial firms often find themselves grappling with the question of whether to build, maintain and expand their own solution, or explore outsourced providers. While the prospects of maintaining technology in-house and controlling costs may seem appealing, proprietary solutions require significant investments in operations and infrastructure. Moreover, the technology infrastructure required to effectively work with market data is complex, demanding a high level of resiliency and redundancy to support mission-critical downstream processes. These factors collectively contribute to a firm’s overall total cost of ownership or TCO.
In this blog, we take a closer look at the challenges involved with capturing, storing and managing accurate and comprehensive tick history data and why, with so many factors to consider, it can be such a costly undertaking to get it right. We will also explain why our Tick History – PCAP solves many of these challenges, providing an efficient and superior TCO solution that still offers the depth of market detail your business needs.
With our Tick History — PCAP and AWS S3 Direct offering, firms can reduce their monthly total cost of ownership by as much as 80%.
Adding up the costs: the key components for market data capture
Connectivity: Firstly, establishing low network latency connectivity is paramount. Capturing the highest quality market data requires capturing it as it flows through the exchange with minimal network hops and utilising high-performance network infrastructure (switches, routers, etc.) to minimise the total network latency. This often requires co-locating with a network provider or within an exchange’s dedicated co-location facility. Adding new venues requires capacity planning and bandwidth upgrades, further increasing the infrastructure footprint.
Hardware and capture: Hardware and capture solutions play a vital role in creating a highly resilient market data infrastructure. Redundant capture servers – fitted with high-performance network interfaces, switches, timestamping servers and devices are necessary to ensure a reliable and resilient market data capture set-up. Timestamping is one of the critical processes here and requires using a high-quality time-source (GPS or similar), time synchronisation (such as PTP, White Rabbit and PPS) – to improve the accuracy of timestamps on packets. As firms look to establish a global market data footprint, the costs to scale up this infrastructure grow exponentially. Lastly, firms also need to build a capture solution, to capture the market data as it is received by the network interfaces, split, and process the packets containing market data.
Monitoring and alerting: Operating a global market data infrastructure requires a robust monitoring and alerting stack to identify and mitigate issues in a timely manner. Numerous metrics including server health, network health, time accuracy and software performance are monitored to maintain a 24×7 live capture infrastructure. Every instance of degradation of data quality needs to be identified, then the root cause addressed, to achieve a truly high-performance solution. Also, once captured, market data requires thorough analysis to identify missing or incorrect data, and arbitration between multiple sources, to create a clean and lossless data stream.
Exchange updates and maintenance: Exchanges often upgrade or modify market data feeds and connectivity requirements for various reasons, such as protocol updates, new or updated multicast groups and bandwidth upgrades. Firms are required to monitor these changes, and trigger software or infrastructure upgrades to ensure business-as-usual continuity. This includes coordinating with exchanges and participating in the User Acceptance Testing conducted by exchanges, network connectivity providers and capacity planning. Continuous maintenance increases costs, requiring a global team of analysts, software, data and infrastructure engineers, and market data SMEs to build and maintain the data infrastructure.
Storage: Market data rates keep increasing and, because of the recent increase in high-volatility days in financial markets, firms can now expect to see a YoY 20-40 % increase in the amount of global market data published on the networks. Once captured, market data needs to be sourced, and then stored on-prem or in the cloud for further analysis. As data volumes explode, data automation is rising and more customers are leveraging the cloud to increase operational efficiencies, lower technology costs, and use cloud computing power for data analysis. The sheer volume of data transfer from the capture sites to the final storage location could go up to as much as 700TB per month, increasing storage costs exponentially. Though cloud storage seems an attractive obvious choice, storing historical market data in the cloud is by no means free. Beyond initial costs, there are additional ongoing costs such as market data licensing and operational costs to upgrade systems to handle exchange-mandated updates incurred to capture data.
Lowering TCO with Tick History – PCAP: a high-quality capture and storage solution
TCO: In-house vs Tick History — PCAP S3 Direct
We analysed the TCO for capturing and storing eight of the top global futures exchanges (HKFE, KRX, ICE, CME, EUREX, Euronext, OSAKA, and TAIFEX) and identified that with our Tick History – PCAP and AWS S3 Direct offering, firms can reduce their monthly TCO by close to 80%. Savings based on one-time and ongoing storage costs.
More about Tick History – PCAP
Tick History – PCAP is a cloud-based, 20+ petabyte repository of ultra-high quality global market data, captured directly in the exchange data centers. It enables you to access complete sets of lossless, top-of-book and full-depth data (Levels 1, 2 and 3) in your choice of format.
Our focus is to ensure that the data our clients consume is of the highest quality. Our capture process starts with our Real-Time – Ultra Direct suite of feed handlers and applications deployed to capture, monitor, arbitrate and normalise the market data powering our flagship Tick History – PCAP solution. We have spent years refining our technology and methodology to handle this difficult and resource-intensive work, so that our clients don’t have to. The result is superior data quality for improved decision-making and optimised performance. Also, with our AWS S3 Direct offering, customers can leverage the full potential of the data in cloud, by accessing PCAP data stored in our AWS buckets as their own, or even bringing their queries to our cloud. Our shared storage fee cost is a fraction of the total cost of the feed itself, based on volume of feed, and is not tied to escalating volume. It remains constant over the year, even as volumes and storage costs rise, so you can keep your costs down and focus on advancing your business.
Note – The TCO costs were computed based on the following costs a) 2x servers for capturing market data b) Network Connectivity and maintenance costs c) Time Stamping & Time keeping device costs e) AWS Storage costs (Standard AWS S3 storage pricing)
Republication or redistribution of LSE Group content is prohibited without our prior written consent.
The content of this publication is for informational purposes only and has no legal effect, does not form part of any contract, does not, and does not seek to constitute advice of any nature and no reliance should be placed upon statements contained herein. Whilst reasonable efforts have been taken to ensure that the contents of this publication are accurate and reliable, LSE Group does not guarantee that this document is free from errors or omissions; therefore, you may not rely upon the content of this document under any circumstances and you should seek your own independent legal, investment, tax and other advice. Neither We nor our affiliates shall be liable for any errors, inaccuracies or delays in the publication or any other content, or for any actions taken by you in reliance thereon.
Copyright © 2023 London Stock Exchange Group. All rights reserved.
The content of this publication is provided by London Stock Exchange Group plc, its applicable group undertakings and/or its affiliates or licensors (the “LSE Group” or “We”) exclusively.
Neither We nor our affiliates guarantee the accuracy of or endorse the views or opinions given by any third party content provider, advertiser, sponsor or other user. We may link to, reference, or promote websites, applications and/or services from third parties. You agree that We are not responsible for, and do not control such non-LSE Group websites, applications or services.
The content of this publication is for informational purposes only. All information and data contained in this publication is obtained by LSE Group from sources believed by it to be accurate and reliable. Because of the possibility of human and mechanical error as well as other factors, however, such information and data are provided "as is" without warranty of any kind. You understand and agree that this publication does not, and does not seek to, constitute advice of any nature. You may not rely upon the content of this document under any circumstances and should seek your own independent legal, tax or investment advice or opinion regarding the suitability, value or profitability of any particular security, portfolio or investment strategy. Neither We nor our affiliates shall be liable for any errors, inaccuracies or delays in the publication or any other content, or for any actions taken by you in reliance thereon. You expressly agree that your use of the publication and its content is at your sole risk.
To the fullest extent permitted by applicable law, LSE Group, expressly disclaims any representation or warranties, express or implied, including, without limitation, any representations or warranties of performance, merchantability, fitness for a particular purpose, accuracy, completeness, reliability and non-infringement. LSE Group, its subsidiaries, its affiliates and their respective shareholders, directors, officers employees, agents, advertisers, content providers and licensors (collectively referred to as the “LSE Group Parties”) disclaim all responsibility for any loss, liability or damage of any kind resulting from or related to access, use or the unavailability of the publication (or any part of it); and none of the LSE Group Parties will be liable (jointly or severally) to you for any direct, indirect, consequential, special, incidental, punitive or exemplary damages, howsoever arising, even if any member of the LSE Group Parties are advised in advance of the possibility of such damages or could have foreseen any such damages arising or resulting from the use of, or inability to use, the information contained in the publication. For the avoidance of doubt, the LSE Group Parties shall have no liability for any losses, claims, demands, actions, proceedings, damages, costs or expenses arising out of, or in any way connected with, the information contained in this document.
LSE Group is the owner of various intellectual property rights ("IPR”), including but not limited to, numerous trademarks that are used to identify, advertise, and promote LSE Group products, services and activities. Nothing contained herein should be construed as granting any licence or right to use any of the trademarks or any other LSE Group IPR for any purpose whatsoever without the written permission or applicable licence terms.