Big Data - Are You Just Weathering the Storm or Getting Ahead?

Chris Patti, CTO, AccuWeather

Weather is the original Big Data pioneer, with its vast and complex data points, measurements, and variables from around the world. Abstracts from ship logs, tracking weather conditions collected by the Smithsonian meteorological project, date back to 1796. Imaging hundreds of officers on hundreds of ships independently collecting weather data – with no way to share neither the data nor compile or compare information.

Today, times have changed significantly. Big weather data is modeled, aggregated, and shared effectively on an exponentially larger scale. Terabytes of data are produced every day from hundreds of weather sources and IoT is the next frontier in producing massive amounts of data. To turn data into truly actionable information requires analytics and computing power with the Cloud used as a scalable and reliable data resource. Using the computing power of the Cloud, AccuWeather can turn raw weather data into actionable insights that people can use every day, to make the decisions that improve and protect their lives.

A Superstorm of Data

For consumers, there is an inherent desire for more weather information right here, right now. When consumers request weather data, they may have a commute, a weekend, a vacation, or a home at stake based on the information they seek. They call up their favorite weather app on their mobile device and expect to see the information that matters most to them, whether it’s the current temperature, minute-by-minute precipitation forecasts, crowd-sourced weather and road conditions, or even a look ahead for the next 25 days, and they want it now – without room for lag.

AccuWeather is the world’s largest weather media company, serving over 12.5 billion data requests per day in over 100 languages and dialects, on every popular platform and device. This kind of data heavy lifting takes computing power that can not only handle the tremendous increase in global demand, but also can flex and expand with seasonal and event-driven data spikes, in which severe weather events regularly break lifetime records of demand.

For enterprises, that demand is multiplied exponentially. According to estimates, the volume of business data worldwide, across all companies, doubles every 1.2 years and it is only increasing. AccuWeather Enterprise Services, which works with over 240 of the Fortune 500 companies and thousands more, recently introduced Data Driven Decisions (D3) Analytics which is helping businesses create new levels of actionable business intelligence.

Establish a clear strategic data architecture, infrastructure, and data support platform that will allow to proactively provide accurate, trusted, and insightful information to users

Retailers are now comparing SKU level product information with historical and predicted weather data, and working with our data analysts to create data adjacencies that will show how even slight variations in weather will impact the sale of a particular product on a daily basis, and even projected over the course of a seasonal forecast. Armed with clear insights, they can better manage stock, execute in-time logistics, and better market on a store-by-store basis.

Likewise, healthcare providers can create and maintain readiness for the most statistically probable health conditions and energy producers and brokers can anticipate demand. The possibilities are as myriad as the business needs of each client and enterprises are only beginning to understand the opportunities – all thanks to the ability to manage data efficiently and effectively.

A Common Challenge

The challenge of managing global data and its growth is not uncommon. In fact, AccuWeather grew from 4 billion data requests per day to over 12.5 billion in just three years as a result of continued growing demand for all-weather content, and it is increasing on a daily basis. The weather-information-seeking-audience accesses content through virtually any connected device, including smartphones, tablets, and computers but also connected cars, smart appliances, and wearables. The data is available through both proprietary and third-party web sites, mobile web sites, and an ever-growing list of social media. The list just keeps growing and so does the nature and number of demands on those computing capabilities.

For these reasons and more, AccuWeather migrated its own API from an on-premises datacenter to a cloud-based platform in 2012. Leveraging a flexible framework, we handle billions of requests each day and can see when, where, and how often our weather data is being used. For example, it used to take four days to process one day’s worth of log information; now, it is now completed in near real-time. By understanding when and where users are likely to request data, on what devices, and the type of data they need, AccuWeather can ultimately create a superior weather experience.

The Sky is the Limit for Innovation

Weather provides another unique application for cloud computing. Weather is ever-changing, and fluctuations in weather data traffic are uniquely impacted by severe weather events, holidays and more. For that reason, it’s hard to anticipate spikes in usage. For AccuWeather, cloud computing has provided the power of adaptive scalability, and unprecedented flexibility to meet spike demands and shrink when it’s no longer needed.

Though there has always been a tremendous amount of data gathering and advancement in meteorology, there are more data sources available now than ever before, and the applications for their use are limitless, across businesses and consumer applications worldwide. For AccuWeather, cloud computing allows for the flexibility and scalability to integrate new global data sources, including radar, satellite imagery, numerical weather forecast and weather sensor information, and more - to deliver the customized weather information that customers need to improve their lives.

To stay ahead, based on AccuWeather’s experience and expertise, do not reactively weather the storm – changing and adapting slowly based on big data demands. Establish a clear strategic data architecture, infrastructure, and data support platform that will allow you to proactively provide accurate, trusted, and insightful information to users, delivering true value and impact.