广州广重社区打造"1234"模式创新社区志愿服务新载体

Today, change is afoot in Poland's scientific research community, as the availability of unprecedented amount of data reshapes the scientific process. As research institutions navigate the uncharted water, they are confronted with an array of challenges:

• How can they effectively manage the exponential growth of data while ensuring efficient storage and retrieval processes?

• What strategies can be employed to develop robust data lake solutions capable of handling diverse data formats?

• In what ways can storage and computing systems be optimized to process and analyze scientific research data in real time, keeping pace with the speed of data generation?

These are the questions Poland hoped to answer through its National Data Storage (NDS) project. The project was initiated with the aim of making groundbreaking improvements to Poland's "data infrastructure for sharing, storage, and effective processing of large data volumes in high-performance computing (HPC), big data and artificial intelligence (AI) models".

And that is exactly what Poznan Supercomputing and Networking Center (PSNC, or PCSS in Polish), a key role in the NDS project, focuses on. Affiliated to the Institute of Bioorganic Chemistry of the Polish Academy of Sciences, PSNC is one of the top-tier supercomputing centers in Poland and beyond. Its vision lies at the core of everything it does: striving for "integration and development of information infrastructure for science" and trying to be everywhere science meets business and innovation affects society.

In Deep Water

Supported by an array of high-performance storage devices, PSNC expanded its functionality of the supercomputing center and provided users, mostly universities and R&D institutions, with premium HPC and data lake services. These research institutions were increasingly focused on high-performance data analytics (HPDA) and AI, and therefore expected PSNC to process massive amounts of unstructured data. Research institutions working in fields like genome sequencing, satellite remote sensing, supercomputing, big data, and cloud computing were in urgent need of real-time, efficient data processing and analytics.

PSNC was no stranger to such requirements. However, they were facing a host of tricky problems on their live network: IT system devices like storage ran at full capacity and consumed a lot of power; multiple IT devices from different vendors made operations and maintenance (O&M) and management difficult and costly.

Plus, in pursuit of a rock-solid HPC platform for the NDS project, PSNC knew that it would have to go beyond mere computing speed; it would also need to analyze exabytes of data, and this would require a comprehensive upgrade of its data storage infrastructure.

Fueling HPC in the Wave of Mass Data

Huawei, with its long record of building solid data infrastructure for various industries, was well placed to meet these requirements. That can be seen in the scale-out architecture of the solution Huawei proposed to PSNC. Huawei identified its OceanStor Pacific storage systems as an excellent match for the architecture, given PSNC's need for an ultra-high density, hybrid load-oriented design, and their use of multiple protocols. The scale-out storage systems would support high bandwidth and IOPS-intensive workloads, delivering efficient and reliable services for the scientific community that PSNC serves.

Storage devices are the foundation of any storage system, and choosing devices that are fit for purpose is critical. But this is also true for tools that help to unleash the power of HPC. The distributed parallel client (DPC) is such a helper that is designed by Huawei in HPC scenarios. It would provide PSNC higher single-client and single-stream performance to fit the service needs.

Building such a solid infrastructure, however, was no easy feat — it required more high-performance storage systems to suit specific service requirements, and that's where Huawei's OceanStor Dorado all-flash and OceanStor hybrid flash storage systems step in. Purpose-built for the Lustre architecture, these storage systems would provide remarkably high input/output operations per second (IOPS), significantly outperforming the performance delivered by the legacy devices used at PSNC.

In PSNC's HPC system, network design is critical in addition to computing and storage capabilities. A lossless network with high performance and low latency is key to the decision-making of PSNC. Huawei provided also the data infrastructure network solution using CE series switches, offering high-performance and reliable network connections. The solution would also offer end-to-end RoCE Ethernet, exclusively achieving zero packet loss while doubling the computing power compared with traditional Ethernet. Furthermore, considering the future cloud-based evolution of PSNC, the solution would provide an open Ethernet network to support the cloudification of computing power and improve the scheduling efficiency. PSNC gave Huawei's proposal the thumbs up and Huawei technicians soon arrived in Poznan to implement the solutions.

Since the deployment of these systems, PSNC has strengthened the data throughput to HPC platform in the NDS project via Huawei Storage equipment. PSNC now streamlines data analytics and management processes, saving a lot of time, costs, and resources. In the words of Norbert Meyer, the Director of HPC & Data Division PSNC, "The Huawei solution allows us significantly to improve the data infrastructure in HPC and cloud environments, used for scientific and industry purposes. When we tested, we were positively surprised with the stability and reliability of the storage systems."

Navigating the Future with Huawei

From the shipyards of data infrastructure planning to the high seas of scientific innovation, PSNC and Huawei have gone through a lot in their decade-long partnership, and they know that the next technical and scientific breakthroughs are just over the horizon.

PSNC is a steward of the dreams and aspirations of scientific researchers in Poland and beyond. Huawei knows the treacherous waters PSNC must navigate on its digital journey.

Together, they will continue to craft data infrastructure that embraces AI, Big Data analytics while also contributing to a low-carbon future for Europe.