As economies make the transition to green, renewable sources of electricity such as wind and solar, engineers, economists, energy producers and consumers, and regulators are struggling with how to integrate these new energy technologies into the grid. A major challenge is that wind and solar are only intermittently available and their availability may not coincide with periods or locations of high demand. 

To address the topic of electrification via renewables, the Institute for Mathematical and Statistical Innovation (IMSI) hosted a Long Program on “The Architecture of Green Energy Systems,” during the summer of 2024. Professor Frank Wolak of Stanford University spoke about the array of challenges in integrating wind and solar into the grid, primarily from the perspective of energy markets and regulation. Interestingly, making progress on this issue involves a complex interplay between engineers, energy sellers and economists, and regulators, all of whom must engage on a playing field that is ultimately bounded by nature. The constraints are not only whether the sun will shine or the wind will blow, but also the immutable laws of physics – Ohm’s law and Kirchoff’s law – that govern the behavior of electricity and the physical characteristics of the grid. 

Wolak summarized the basic challenges with wind and solar energy. First, in addition to their inherent intermittency, grid scale (as opposed to, say, roof-top solar panels for home use) wind and solar electricity generation must be located in areas that have consistently high winds or a reliable minimum number of sunshine hours in a year. And typically, at least in the US, those areas large enough for reliable grid scale production are not close to population centers. Therefore, integrating these new sources of electricity into the grid requires construction of new networks of transmission lines, which are expensive and also inefficient over long distances.

Second, the electricity market is vastly different from the market for, say, cars or gasoline. This difference is mainly due to the fact that a) electricity production and distribution are done in many parts of the US by regional utilities (many of which are a local monopoly), b) the distribution of electricity occurs over the grid, and c) the basic price for electricity is largely set by regulations. In other words, there is little to no competition that can drive consumer choices, meaning, for example, a family cannot shop for their electricity in the same way they might shop for a car or choose to drive an extra mile to get cheaper gasoline.

Wolak approaches this important topic that lives at the intersection of many fields and that has real-life implications for keeping the lights on, particularly during extreme events that place unusually high loads on an electrical system, from the perspective of a professor in the Stanford University business school. The introduction of wind and solar, with their inherent intermittency, either predictably periodic (day vs night) or random (cloudy days, no wind), can tend to exacerbate the instabilities of an electrical system already straining at capacity. It is therefore critically important that regulators, energy marketers, and plant operators – and the students studying to pursue these careers – understand the rules, interactions, dependencies, and sources of instability inherent in an electric power system. To respond to this need for understanding, Wolak and his colleagues developed a computer game called the “Energy Market Game.” Wolak has found that gamifying his course on energy markets is pedagogically powerful, particularly since his students come from a variety of educational backgrounds, professional interests, and technical knowledge. Introducing an element of competition into what is at its core a rules-based system with an overlay of randomness and various levels of risk and uncertainty, takes this important topic out of the theoretical classroom and allows teams to learn practical knowledge and skills within the game. Indeed, Wolak and his students have found the Energy Market Game to be so effective that it’s now also used for training in the real world of energy markets.

To provide further insights into how green energy can be integrated into the electrical power system, Pascal Van Hentenryck from Georgia Tech, talked about how machine learning and artificial intelligence systems can be important adjuncts to the tools already in use. Van Hentenryck discussed how the traditional electric power paradigm, in which electricity is produced at a central source, then delivered to consumers via the grid, is being transformed from its traditional consumption model to a more modern prosumption model. Prosumption, in the context of the electrical grid, means that the consumers of electricity are also producers of electricity.  For example, a household with rooftop solar or a plug-in electric vehicle could be in a position to sell surplus electricity back into the grid, either in the middle of the day when the sun is shining but nobody is home, or overnight, when their EV is plugged in. Consequently, this multi-directional flow of electricity into and out of the grid complicates the economics of selling and buying electricity. 

For most utilities, the electricity market consists of a “day-ahead” market and a “real-time” market. The day-ahead market is where most of the activity occurs, and allows sellers and buyers of electricity to settle on a price based on projections of the next day’s supply and demand. The real-time market operates on a one-hour to five-minute time scale, and allows producers and consumers to make fine adjustments to the supply and price of electricity according to changes in the load (demand) that occur in real time. Think of the increased demand for air conditioning on late August afternoons in Houston, just as the sun is setting and people are getting home from work to cook dinner. This is a great example of spiking demand just as solar is going off line for the night.

Van Hentenryck’s group is developing AI tools to set prices in the day-ahead and the real-time markets. Given the importance of these tools for peoples’ general well-being – the lights need to stay on; people need to cook and find relief from the heat or cold outside – Van Hentenryck’s team places high value on a number of key factors that cannot be compromised: a) trustworthiness – “the system’s ability to ensure reliability and performance, and to inspire confidence and transparency”, b) explainability – “ensuring that AI decisions are understandable and accessible to a large diversity of participants”, c) confidentiality, particularly in terms of data privacy, for those engaging in the markets and for end users, and d) computational speed of teaching the AI and then implementing the tool in real time. 

In practice, utilities have a series of optimization problems to solve in deciding which electricity generators they need to switch on or off at a particular time and how much power each generator will produce when it is on line. These decisions are made based on predictions of future demand at various timescales (next day, 15 minutes from now, 5 minutes from now), the availability and price of resources to generate the electricity (e.g., oil, coal, gas, wind, solar, nuclear), and then bids on the price of electricity produced by each of the generators at the time they are deployed. These factors make up a complex, multi-layered feedback loop which is further complicated by real-time conditions that may differ from the assumptions made during the day-ahead market process. Figure 1 is illustrative of the complex interdependencies involved in arriving at a pricing decision (green) given bids and other input data (red), optimization constraints such information need to make a day-ahead (DA) or real-time (RT) market estimation (blue), and externalities such as the availability of electricity from renewable sources (purple).

To create an AI system that works for utilities, Van Hentenryck and his colleagues need to design a tool that minimizes risk under a set of empirical physics, engineering, and business constraints inherent to the system. The tool needs to be trustworthy by design and reliably deployable in actual critical infrastructures. Finally, it needs to meet minimal performance expectations in terms of the quality of its solutions, and it needs to be efficiently scalable up to the size of the grid and the market.  Figure 2 illustrates the machine learning process that includes physics-driven constraints due to Ohm’s law and Kirchoff’s law.

One important outcome of Van Hentenryck’s work on applying AI and machine learning to the grid is an optimization-based risk assessment ML tool to predict the probability of congestion on each power line in the grid. Their tool is able to generate feasible solutions based on real-time conditions that minimizes financial risk to the utilities. And of course, the lower the financial risk to the provider, the higher the probability is that the consumer will have a reliable flow of electricity to their home at a predictable and stable price.

The talks by Wolak and Van Hentenryck illustrated the complexities of introducing green renewable sources of electricity into the grid from a variety of physics, engineering, and economic conditions that involve interconnected decisions and external conditions. While Wolak is developing methods for understanding and responding to the increasing greening of the electrical grid, Van Hentenryck is focussed on applying machine learning and artificial intelligence to the number of decisions that need to be made to keep the grid operating in real time. Both talks were illustrative of the impact of IMSI’s Long Program on “The Architecture of Green Energy Systems,” particularly the importance of bringing mathematical scientists together with experts from allied fields so they could focus on understanding and developing solutions for some of society’s biggest challenges.