I'm trying to tackle a very challenging problem and I would appreciate your help.
My organization has a lot of different items which can be demanded by our clients. Those items can also be returned back to our warehouses. Our goal is to predict how many instances of each item our clients (which are actual warehouses) will demand next year.
As for now, my organization is predicting the demand using a weighted average, based on the quarterly demands of the last 7 years. This gives us a prediction for the demand for the next quarter. Then, they multiply this prediction by 4 to get the next year's prediction. This is my baseline to beat. Our data is irregularly sampled, meaning that demand can occur any day. This also means that we have to resample the data ourselves (we tried monthly and quarterly).
Each demand has a few features:
- Item Id
- Date of Demand
- Receiving Warehouse - our organization has a lot of warehouses spread out across the country, this is the warehouse that received the item.
- Sender Warehouse - the warehouse that sent out the item.
- Type of Demand
- Quantity - can be either positive (if the client consumed the item from the warehouse) or negative (if the client returned the item back to the warehouse)
Again, for clarification: the goal is to predict the Total Quantity of the next year for each item. Obviously, this is a time-series forecasting situation. The problem is that we have over 20K different time series (because each item’s history is a distinct time series).
What we have done so far:
Simple statistic models (exponential smoothing, Holt-Trend Methods, etc.)
Classical machine learning models (linear regression, decision trees, etc.)
Ensemble methods (random forest, boosting, etc.)
Simple deep learning models – neural networks with a few linear layers and an RNN layer.
Complex state of the art neural networks for time series forecasting (we tried Google's Temporal Fusion Transformer)
Calculating the regular weighted average (as the company does now) but only after removing anomaly demands (based on a DBSCAN algorithm).
The main problem is that none of the above methods can beat the regular weighted average for all the time series we have – meaning that for many of them the new proposed model preformed even worse than the weighted average. Also, we couldn’t find any pattern that allowed us to match each time series to a "winning" method.
What we are looking for is a method which will perform better for all the different time series.
Any help will be highly appreciated.
Thanks in advance.