Why a Good Forecast Matters

Widgets: The Primer

Imagine you’ve identified a great opportunity to import and sell widgets in your country. There’s demand, a reliable supplier, and you’re the first to jump on the opportunity. Here’s how your business might unfold:

You begin by purchasing 100 widgets from your supplier on January 1st. They arrive two months later, on March 1st, and you sell all 100 in just one month. Having successfully tested the market, you place an order for 200 more on April 1st. While you wait for delivery, there’s a temporary lull in sales, but you’re just getting started.

On June 1st, your 200 widgets arrive. By the end of the month, you’ve sold 150, leaving you with 50 units in stock. You order another 200 on July 1st, but your stock runs out mid-July, leaving you unable to sell anything until your new shipment arrives in September. This time, you order another 200 units right away on September 1st, ensuring you’re not caught without stock again.

However, sales slow down in September and October. It turns out that demand for widgets peaks earlier in the year, and you only manage to sell 50 units each month. By November 1st, you’re sitting on 100 unsold widgets, and 200 new ones arrive, pushing your stockpile to 300 widgets. With the slow sales cycle, you’re unsure when or how you’ll be able to sell them.

By the end of the year, you’ve sold a total of 450 units, but you’re left with 250 unsold units—highlighting the challenge of managing inventory, understanding market cycles, and aligning supply with demand.

Figure 1: The timeline of widget sales and supply

This is a (very) simplified picture of what it’s like to manage a supply chain, but it serves to illustrate an important point: without an accurate forecast of your sales, you can end up short of stock (and lose out on possible revenue) or overstocked (and have no free cash left to do things like hire sales people). This is why demand forecasting is such an important part of managing a supply chain.

So what is demand forecasting, anyway?

Simply put, demand forecasting is the science (and sometimes art) of predicting the sales of a product in the market, usually weekly or monthly. In the widget example above, you might consider a monthly demand forecast. If you were very good at understanding your demand, you might predict the following sales when you do your first forecast on 1 January:

This forecast allows for a slow ramp up in January, assumes that demand spikes in June, and bakes in the “market knowledge” that widgets sell less in the second half of the year.

Based on this forecast, you decide that you will order every month just enough to meet the demand two months ahead using the forecast. This means you will order 100 units in Jan (for Mar), but then another 100 in Feb (for Apr), and another 100 in Mar (for May), and so on. Notice in the scenario below that the forecast isn’t always accurate. For instance the actual sales in April is only 50, whereas the forecast was 100. Because of this, you will adjust your order in May (for July) down by 50 units (down from 100 to 50) to prevent going over stock.

Figure 2: The timeline of widget sales and supply, with forecasting

In this scenario, the stock at the end of the year is only 50 units, and we’ve managed to sell 700 widgets in the year. That’s 55% more sales with less than a quarter of the residual stock at the end of the year!

An even better forecast than this would unlock further sales still - it’s possible in August that there’s demand for more than 50, but we just ran out of stock so couldn’t sell more than 50 units. Other months where the closing stock is 0 could have the same issue. Furthermore, we ended on 50 units and had a maximum closing stock of 100 in the year, due to over-forecasting in some of the months. So a better forecast could help reduce the “working capital” required to run your widget business by keeping the unsold stock down.

It’s clear that using a demand forecast to inform your buying strategy can dramatically improve your sales and keep down your costs! And that’s not all - sophisticated forecasts can help with price setting, provide insights on how to gain market share, suggest optimal promotions, and more. 

But all of this only works if the forecast is accurate. Inaccurate forecasts can cost a supply chain company millions of dollars, and it can be extremely difficult to improve the accuracy of forecasts for a number of reasons.

Forecasting Blues

One of the challenges with accurate forecasting is that humans are notoriously bad at statistical reasoning. Daniel Kahneman covers this phenomenon in detail in his excellent book “Thinking Fast and Slow”. It’s difficult for a person, however familiar with the product or customer, to provide an unbiased numerical prediction of future sales. Another challenge is that in the real world you usually don’t sell to just one customer, and you usually don’t just sell one widget. A large supply chain company might sell 1000 SKUs to 1000 customers. That’s 1 million different (SKU, customer) pairs to track, and each one should theoretically be forecasted, which can be prohibitively time consuming to manage and track, and impossible to devote enough attention to each regardless of team size.

Even when automating some of this, it can be tricky to account for all your marketing activities in a forecast: a simple statistical forecast doesn’t know about the promotion you are planning with your biggest customer for next month. And if you manage to include this information in your forecast, you might not account for the sales boost you get because of a major concert in your city, or the viral youtube video that caused people to fall in love with your product, or the aggressive pricing of your competitors that might cause you to lose market share. 

A good forecast needs to account for market movement, seasonality, holidays that move around (like Easter or Black Friday), events (like customer birthdays, concerts, major sporting events), growth, marketing activity, promotions, the competitor landscape, and sometimes even the weather (which itself is notoriously difficult to predict) - and it needs to weigh all these things in just the right way to avoid systematic bias in its predictions. It requires all parts of your business to communicate, with data, about all of their activities. It can be extremely daunting to get started, and many companies experience poor forecast performance and/or high costs in maintaining their forecasts, both manual and automated.

A bad forecast can be worse than no forecast at all - in addition to being expensive to maintain, it can lead to all the issues it’s supposed to prevent, like overstocking and underselling.

Can you avoid forecasting?

Demand forecasting is a lot of work, and when it goes wrong it can be very risky, so it’s natural to ask - can we get away without a demand forecast? There are some options available, and we will briefly explore them here.

One way to “cheat” is to agree sales volumes with your customers in advance. This isn’t exactly going to replace a demand forecast, but a “customer provided forecast” may be more accurate, if customers can be relied on to stick to their word. This doesn’t always work - your customers care about their inventory and sales just as much as you do, and are unlikely to stick to a committed volume unless they sell what they thought they would - i.e. they now need an accurate forecast in order for this to work, so the forecast didn’t actually get removed. 

The main reason you need forecasting in the first place is because there’s a lead time to get stock - if you can reduce your lead time, you will not need to predict your sales as far out into the future. In the extreme case, you can reduce your lead time to almost nothing - e.g. if you built a widget factory and could produce a new widget within 1 day instead of waiting 2 months for them to arrive, you could wait for your customers to place an order, and then manufacture widgets to precisely match their order each time. Setting aside the difficulty of practically implementing a lead time of 1 day, this still doesn’t completely solve your problems. Typically your factory will need to purchase raw materials, some of which will come with a lead time that requires forecasting, bringing you back to square one. Furthermore, If an order comes in that exceeds your factory’s capacity you may still need to turn it down unless you ramped up production in advance to account for the demand. Moreover, you may occasionally experience breakdowns or require maintenance on your factory. All of these factors essentially mean that you will still need a forecast to operate efficiently, but maybe you can forecast 1 month ahead instead of 2 months, which might mean your forecast will be more accurate and your performance will be better.

Practically, it is unlikely that most suppliers can realistically expect to “build to order”. There are, however, some other tricks worth looking into. The most interesting is known as “demand sensing”. Imagine that your biggest widget customer is a retailer called “Retailer X” (strange name, maybe it’s owned by Elon Musk). If Retailer X sells 10 widgets this week, they’ll order them back from you next week. So you convince them to share their sales figures with you every week - this gives you a signal of what your sales to them will be, one week in advance - kind of like a forecast. If you buy or produce widgets when Retailer X sells them, you’ll always be a week ahead of you having to sell them, so if your lead time can be reduced to a week, you won’t need a forecast. Again it’s not perfect - Retailer X may want to build stock for a promotion and surprise you by buying more than they sold, and not every retailer will be as willing as them to share data. Practically, using demand sensing as an input to your demand forecast will yield the best results.

Another tactic to reduce the reliance on highly accurate forecasts is to implement optimal inventory management. This is the process by which you build up so-called safety stock - extra inventory to address any uncertainty in your sales (and also in your supply chain, e.g. uncertain lead times). By buying a bit of extra stock each time, you can build up a buffer that lets you sell beyond what you predicted. There are algorithms that will compute the optimal ordering for you. These algorithms require a forecast as input, but they mitigate the risks of buying too closely to a forecast that may have errors in it, by trading lost sales opportunities against extra inventory. Typically you will be able to sell more by using this. A by-product of these algorithms is that they can be used to quantify the impact of improving your forecast - they can tell you for instance that a 1% improvement in forecast accuracy will reduce your inventory by $100k and improve your sales by $50k in your particular case.  

These tactics - reducing lead times, agreeing volumes with customers, demand sensing, and optimal inventory management - all help improve your sales and keep stock under control, but they don’t remove the need for a demand forecast. Sometimes they’ll enable you to survive with a shorter forecast, and sometimes they will amplify the effects of a good forecast throughout your supply chain, making you more efficient than ever. But a good forecast is still an unavoidable ingredient in an efficient supply chain business. 

Our demand forecasting story - Hudson

Isazi is the proud creator and owner of Hudson, one of the world’s leading supply chain optimization solutions. The product serves some of the world’s largest supply chain customers, including Unilever and Arçelik, and we have built out a comprehensive solution to manage every aspect of the supply chain, from forecasting to inventory optimization to promotion optimization to vendor managed inventory. But demand forecasting is where it all began, almost 10 years ago.

We were approached by a major FMCG supplier, who had over 2000 active SKUs that they were selling to more than 10,000 different customers. That’s over 20 million forecasts they needed to produce on a weekly basis - each one for 104 weeks making the total number of predictions required over 2 billion. Since the forecast was being done by humans, there was no way to do it accurately - there were lots of spreadsheets and lots of assumptions. They wanted us to help them with their demand forecast to see if AI could help them do better. Our policy is always to understand our customers as well as possible so we can give them the best outcomes, so we declined to build a new forecast immediately but instead proposed doing a data and AI strategy for them, mapping out all their data and determining the best opportunities to achieve cost savings and improved revenue using data and AI. They agreed, and we interrogated almost 200 different data sources from their business, and mapped out how data was visible at all the different parts of the supply chain. 

This process surfaced many opportunities to use AI to improve their business, from optimizing promotions and marketing to inventory optimization to production line optimization. But, as is often the case when entering a new domain, we learned from the data what the experts had told us from the start - all of these things relied on an accurate AI powered forecast at their core. So we set out to build the best AI forecast that had ever existed for suppliers and manufacturers.

Our first discovery was the importance of data - you cannot build a good forecast on prior sales alone - you need stock availability, promotion activity, marketing information, competitor analysis, market share studies, key holidays, customer holidays, sales-out data from customer tills, not to mention clean master data. The data had to provide real-time insights into what the business was doing to sell, in a form that our models could understand. To accomplish this feat of data engineering, we built out robots that could pull data from email attachments, from SAP, and from other systems in the business, and we piped it all into our Hudson platform, where we leveraged big data tools to handle the sheer scale of the data that we were collecting.

Our second discovery was that synthesizing all of this information together into a forecast without introducing bias and overfitting is extremely difficult. Typically, you might have 2 years of weekly history for the sales of a product at a customer. That’s just over 100 data points, which is almost nothing. If you want to fit a complex model with thousands of parameters to predict this, it will immediately over-fit to the 100 data points. Worse, there are only two data points on how this model sells over black friday, and maybe only one on how a specific promotion affected sales. Separate models per product had to be very simple to be effective, but if we tried to train a model that could predict multiple products in some way, suddenly there was more data: over 2 years at 2000 SKUs and 10k customers, there are 2 billion data points - a lot more than 100. The details are very tricky, but a combination of methods like hierarchical forecasting and embeddings enabled us to take advantage of these extra data points to fit models that could take advantage of all the data we had collected.

Our resulting forecast almost doubled our customer’s accuracy, and resulted in millions of dollars of savings in inventory in just the first few weeks after implementation. Emboldened by our success, we started building a product that could achieve the grand vision we had outlined in the data strategy. Today, Hudson continues to offer the strongest demand forecast in the market, and also contains other modules that add immense value to our customers, including inventory optimization, stock allocation, promotion optimization, vendor managed inventory, and others. 

We have built strong relationships with some major suppliers who are using Hudson, with proven success in each case. However, each of our implementations have been highly specialized and involved a lot of consulting and specialization to each of our customers’ needs. 

Making all of this available to everyone

Over the years, we’ve seen the massive impact that Hudson has had on our customers, and we’ve asked ourselves, could these benefits be delivered to smaller companies at a lower cost with less intensive consulting? We think the answer is “yes”! We are so passionate about this, because if it is possible to make a significant fraction of the world’s supply chains more efficient, we believe market forces will then drive down the costs of goods - and even reduce the cost of living - and that’s a goal worth shooting for! 

We want to play our part in this, even if it is just a small part, so we’re really excited to announce that we will be launching Hudson Lite in 2024. Hudson Lite will let medium and small businesses take advantage of all of the advanced AI algorithms in Hudson for forecasting and supply chain optimization, at a fraction of the cost. For the first time, truly premium performance should be available to every supplier. This can only be achieved because of AI, which is integrated into the system right from the first interaction you will have with it - to help you integrate your data yourself from any source to feed the models that will give you your results. 

We look forward to the massive savings that Hudson Lite will unlock throughout the world. We will keep ourselves honest by tracking Hudson’s contributions to the global supply chain centrally, and do everything we can to make this number truly transformative.