With Annie Duke, Author of Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts
Podcast: Play in new window | Download
Annie Duke is the author of Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts.
Annie was one of the top poker players in the U.S. She has earned a World Series of Poker bracelet and is the only woman to win the NBC National Heads-Up Poker Championship.
But there’s more! Prior to becoming a professional poker player, Annie was awarded the prestigious National Science Foundation Fellowship and studied Cognitive Psychology at the University of Pennsylvania. Annie also went head-to-head with Joan Rivers to win second place in Season 8 of Celebrity Apprentice.
Annie has mixed her poker insights with cognitive psychology in her new book, Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts. The book is an amazing guide to making better decisions. It’s filled with stories from Annie’s life and offers new ways to change our thinking to get better outcomes in business and in life.
The Worst Call In Super Bowl History
At the outset of her book, Annie Duke tells the story of Pete Carroll, coach of the Seattle Seahawks. Pete made one of the most controversial calls in football history in Super Bowl XLIX.
With 26 seconds left at the end of the fourth quarter, the Seahawks were on the New England Patriot’s 1-yard line. It was second-down, the Seahawks were down by four points, and had one timeout.
Then, instead of making a running play, Coach Carroll told his quarterback to pass the ball. The ball was intercepted in the end zone, and the Seahawks famously lost a Super Bowl game that they were on the brink of winning.
Bad Call Or Bad Luck?
Did Pete Carroll screw-up bigtime? Annie notes that the probability of a short-pass interception is below two percent, based on 15 years of NFL data. Moreover, if the pass is incomplete, the clock stops on its own. The ball can then be handed off to the running back for a touchdown attempt, without wasting a timeout.
Turns out, Pete’s call wasn’t a bad one based on the facts he had. The interception was highly unlikely and, unfortunately, cost the Seahawks dearly. Yet, people blamed Pete Carroll as if an interception was inevitable!
Making The Right Calls At Poker
In Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts, Annie Duke notes that as a professional poker player, she had to constantly make in-the-moment decisions based on incomplete facts. Annie did so by embracing the uncertainty, by estimating the probability of various outcomes, and by guessing how her opponents would play the cards they had.
It was the same for the Pete Carroll play. He went for the lowest risk play, but it simply did not work out.
“Resulting”
As Steve notes, our biases lead us to think that if we have a successful outcome, we’ve made a good decision; but if we have an unsuccessful outcome, it’s bad luck. People also tend not to take the blame for their own mistakes and, therefore, do not really move forward in learning how to be more successful.
Annie adds that if you’re judging the quality of a decision from afar, we judge the decision based on results. If it’s a bad outcome, we assume it was a bad decision and vice versa. We ignore how much luck plays a role in the equation.
On the flip side, when judging ourselves, we engage in a self-serving bias. If it’s a really good outcome, it’s because we’re awesome. When it’s a bad outcome, we want it to not be our fault. Consequently, we end up blaming luck and everything else.
In Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts, Annie Duke says these mindsets, “resulting” and “self-serving bias”, make us feel good about ourselves, but keep us from learning from past outcomes.
Learning How To “Be Awesome”
To underscore this point, say you drive drunk and still get home safely. This may lead you to believe you’re a really good drunk driver! But that’s a folly.
Instead, Annie Duke recommends changing the definition of “being awesome”.
Awesomeness should be about working really hard at everything and hoping Lady Luck pulls in your favor. Awesomeness should also include the humility to take mistakes in stride.
When things go wrong, don’t blame others. Instead, analyze your approach and learn from your mistakes. This approach helps improve decision-making going forward.
So, the next time you’ve had a bit too much to drink, you’ll take more responsibility and have someone else drive you home.
The Fallacy Of Averages
Annie likens life to an experiment with rats. First, rats were given food if they pressed a lever ten times. Then, researchers introduced some uncertainty and gave food for an average of ten lever presses. This uncertainty and averaging confused the rats.
So it is with averages in the stock market, adds Steve. Historical returns average about six percent. Yet, strong down-swings could result in huge losses.
Annie notes that successful learning depends on two things: feedback, and the correlation between feedback and decisions.
For example, if you started smoking in 1920, the feedback you got in 1960 was too late to act on. To be actionable, it’s important to get feedback in a timely manner.
Going back to the rat experiment, there has to also be a correlation between the feedback (getting food) and the number of times the lever needs to be pressed to get food. Giving the rat food on an average of ten presses introduces uncertainty and makes it harder to learn from the feedback.
Feedback Loop In Investments
Steve sees this feedback loop routinely playing out in the world of investments. Portfolio managers who do well in their first few years in the market tend to repeat their investment strategies. But if the macroeconomic landscape shifts, their strategies often stop working.
Much like the rats, outcome uncertainty confuses us and impacts our decisions.
The Fix
Annie Duke believes that being smart isn’t particularly helpful. In Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts, she wrote a section titled “Being Smart Makes It Worse”. The fix is to not believe you’re the smartest but to recognize uncertainty and respect it.
Warren Buffett echoes her views. He has said that it doesn’t take a high IQ to be successful in the world of investing. It takes understanding your own behavior and biases and learning from them.
She also recommends seeking unbiased advice from others. As she puts it, recognize the power of getting other people to help you. You’ll always be better off bouncing your thoughts off others, putting your logic to the test with them, and getting their feedback.
To learn more about how to make better, quicker real-life decisions with limited facts, consider reading Annie Duke’s Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts.
Disclosure: The opinions expressed are those of the interviewee and not necessarily United Capital. Interviewee is not a representative of United Capital. Investing involves risk and investors should carefully consider their own investment objectives and never rely on any single chart, graph or marketing piece to make decisions. Content provided is intended for informational purposes only, is not a recommendation to buy or sell any securities, and should not be considered tax, legal, investment advice. Please contact your tax, legal, financial professional with questions about your specific needs and circumstances. The information contained herein was obtained from sources believed to be reliable, however their accuracy and completeness cannot be guaranteed. All data are driven from publicly available information and has not been independently verified by United Capital.
Steve Pomeranz: The book, Thinking in Bets: Making Smarter Decisions When You Don’t Know All the Facts. Now, what makes this book special is that it’s written by Annie Duke. And you may recognize her name as she went head to head with Joan Rivers on the Celebrity Apprentice.
But more importantly, she’s one of the top poker players in the US. She’s earned a World Series of Poker bracelet, and she’s the only woman to win the NBC National Poker Heads-Up Championship. But there’s more! Annie started her career as an academic going for a Ph.D. in cognitive science before she went into poker.
So she mixed those two things together and produced an amazing guide to making better decisions and that’s the book. And welcome, Annie Duke.
Annie Duke: Thanks for having me.
Steve Pomeranz: It’s a great book. It’s filled with so many stories and ways to think and the ways to change our thinking in order to get better outcomes.
But at the outset of your book, you tell the story of Super Bowl XLIX, the Seahawks coach Pete Carroll made one of the most controversial calls in football history. Briefly describe that for us.
Annie Duke: Sure, so it’s the end of the fourth quarter, there’s 26 seconds left in the whole game.
The Seahawks are on the Patriot’s 1-yard line. It’s second down, they have one timeout, and they’re down by four. So it’s important to note here in terms of kind of why there is this reaction to what happened. It’s that there’s a consensus play that Pete Carrol can call which would be a running play, to hand it off to the running back Marshawn Lynch, who’s one of the best short yard running backs in the history of the game. So everybody is expecting a second down, 26-second plus, down by four, then he’s going to hand it off to Marshawn Lynch. Marshawn Lynch is going to try to sort of barrel through the Patriot lines and attempt to score.
Instead, he called at something really unexpected. He calls for Russell Wilson to pass the ball. Russell Wilson does pass the ball. The ball is quite famously intercepted in the end zone. And I really recommend that people go, you can find it on YouTube, go listen to Chis Collinsworth call this.
So the ball’s intercepted, obviously, the Seahawks lose because of this. And Cris Collinsworth is just going, I cannot believe this call. What was he thinking? That call was ridiculous. And then when you looked at the headlines the next day, there were a variety of headlines, most of which were declaring it to be worst call in Super Bowl history.
I think USA Today called it the worst call in football history.
Steve Pomeranz: Okay, so let’s step back and examine what’s going on here. He had to make a decision. He made a decision in a world or a time period of uncertainty. He had only so much time to bring in all the facts that he could, knowing that skill was going to play a part, some luck was going to play a part, and he had to make a decision. He made a decision that had a bad outcome. But was his decision wrong? What do you think?
Annie Duke: Yeah, so I think you actually made a good point of, you pointed out that he only has the information that he has. And the time, which is short, that he has to make that decision.
And then also, there’s a lot of luck in the way that it turns out, right? We know that the ball could be caught, the ball could be just incomplete. It could be intercepted, there could be a fumble. There’s all sorts of different ways that the future can kind of occur there, and there’s only one that can occur.
And in large part, once Pete Carroll calls that pass play, how that actually ends up turning out is pretty much due to luck. Which of those things actually occur? So the question you should really be asking yourself is how often does the ball get intercepted there?
Steve Pomeranz: Yeah.
Annie Duke: Is it 40%? If it’s 40% it’s probably a really bad call. But it turns out that when you actually do the math, it’s somewhere between, above 0% and below 2%.
Steve Pomeranz: Really?
Annie Duke: So it’s that range. Yeah, so if you look over 15 years of data from the NFL, in that short yardage, I think it was a slap play.
But that short yardage play in the red zone, the interception rate over 15 years was around 2%. So can you aggregate across 15 years? I don’t know, the coverage has changed, but let’s just say it’s 2%. So 98% of the time that really horrible result doesn’t occur. And what you’re getting in exchange for making a pass play there is that if the ball is incomplete, if you don’t actually catch it for a touchdown and end the game with a win there, the clock actually stops naturally on its own.
Steve Pomeranz: Which is good.
Annie Duke: Instead of having to burn your timeout-
Steve Pomeranz: Yeah.
Annie Duke: And then you go hand it off to Marshawn Lynch anyway.
Steve Pomeranz: Got it, got it.
Annie Duke: So you still get to run that. So it caused him this really low interception rate, and it was a really, really, really unlikely event.
Steve Pomeranz: Yeah.
Annie Duke: And yet people acted like it was inevitable.
Steve Pomeranz: Well, this is the idea that if your probabilities are 80 favorable and 20 unfavorable, that you’re probably going to win. But someone’s got to be in that 20% category, and that’s unknowable and that’s really part of the risk.
Now you play poker, every hand you’re trying to determine what you know and what you don’t know, and what you can discern from the other poker players across the table. How do you do that so quickly and know that you’re making the right decision and also know that you’ve made the wrong decision?
Annie Duke: Yeah, so in the moment, I think the one thing that’s really important is to sort of get away from this idea of know, right? It’s to really embrace the uncertainty that given the information that I have, this is my estimate of the probability of one or the other things occurring.
And what are the things you are trying to get at there? Well, your opponent’s cards are faced down to you, so you’re trying to figure out sort of what the range of possible hands that they could have is. And the better player you are, the better you are at kind of narrowing down what the possibilities are for your opponent’s cards.
And then once you figure out what your opponent’s cards are, you have to figure out how your opponent is going to react to those cards.
Steve Pomeranz: Yeah.
Annie Duke: That’s really important because one person could have a pair and think it’s the bee’s knees, and another person could have a pair and think it’s a really horrible hand.
So how somebody perceives the hand is going to be different in how they might react to the hand.
Steve Pomeranz: Yeah.
Annie Duke: So you’re trying to sort of plug those variables in, get a guess at that. And then take a look at how often, given these things that I’m sort of guessing at, what’s the probability I think that I’m going to win this hand or lose this hand.
And then figure out if you’re getting the right return on investment. So you kind of get out of this idea of know.
Steve Pomeranz: Yeah, knowing.
Annie Duke: The word, K-N, yeah, knowing. And what you say is, know, I’m estimating, I’m estimating what my probabilities are. And I want to get better and better at that so I can figure out if, given the amount of uncertainty that I have, if I’m going to get enough back.
And you could think about that in terms of the Pete Carroll play. There’s all sorts of stuff that he knew and all sorts of stuff that he didn’t know. Just like at the poker table, he had a time limit on making a decision. He made the best decision that he could given the things that he was guessing at.
About what the Patriots’ coverage was going to be, how much they might be expecting a pass play versus a run play, how good their defense was, all this stuff that he’s estimating.
Steve Pomeranz: Yeah.
Annie Duke: And then he says, well, I think it’s really low probability that the ball is going to get intercepted.
But the fact is, just as you said, I can make a decision that’s going to work out 80% of the time, and if the 20% happens, it doesn’t mean the decision is wrong, it means the 20% happened.
Steve Pomeranz: Let’s get to human nature for a second. So our nature, our biases lead us to think that if we have a successful outcome, we’ve made a good decision, but if we have an unsuccessful outcome, it’s bad luck.
And you didn’t get the promotion, not because it was you, because you did best. It’s because, well, they were a favorite or they, whatever the reason was. People tend not to take the blame for their own mistakes, and therefore, they don’t really move forward in learning how to be more successful.
Annie Duke: Yes, I think this stands in contrast really nicely with what happened with Pete Carrol. So when you’re sort of judging the quality of a decision from afar, right, like I don’t have anything to do with Pete Carrol, you do this thing called resulting. Which is like, if it’s a bad outcome, I assume it’s a bad decision.
If it’s a good outcome I assume it’s a good decision. And you kind of try to make that one-to-one mapping and you ignore how much luck might be in the equation. But what’s interesting is that it stands in pretty stark contrast to what we do for ourselves, which is we engage in this thing called self-serving bias.
Which is, when it’s a really good outcome, we want it to be because we’re awesome. And when it’s a bad outcome we want it to not be our fault. We want it to be like, I can’t believe that happened to me or, that was really unfair, those kinds of things.
And both really stop you from learning.
Steve Pomeranz: Before you go on, we’re going to take a quick break and we’re going to be back learning more about why we have to be so awesome all the time.
Annie Duke: [LAUGH]
Steve Pomeranz: [LAUGH] I’m talking with Annie Duke. We’ll be right back.
Steve Pomeranz: I’m back with Annie Duke. Her book is Thinking in Bets: Making Smarter Decisions When You Don’t Know All the Facts. Annie, we were talking about why we always have to kind of create a story for ourselves where we are awesome. It’s part of human nature as you say.
And actually, when you look at the way we’re wired, that’s actually a good thing even though it can lead to a bad result. I know that sounds confusing. Take us through that.
Annie Duke: I mean I think that, obviously, nobody wants to be depressed really. I mean we all want to think well of ourselves.
So it is important for us to think that we’re sort of good at what we do. The problem is that it depends on whether you’re valuing, like how are you judging whether you’re awesome or not? Are you judging whether you’re awesome in the short run? Or are you judging kind of your awesomeness in terms of achieving your long-term goals?
So if you’re judging your awesomeness in the short run, that you just don’t want to feel sort of the pain of taking on the responsibility for something bad that happened or the pain of a missed opportunity to take on the responsibility for something good that happened. What happens is you’re making these decisions to feel good in the moment.
And it’s really bad for learning. Because we know that all good things aren’t the result of great decision making. It’s some combinations of decisions and luck. And all bad things aren’t the result of bad luck, it’s some combination of decisions and luck. In the simplest sense, think about if you apply that strategy to drunk driving, like you drive drunk you get home safely, hey, that’s a good outcome.
Steve Pomeranz: Yeah.
Annie Duke: I must be a great drunk driver.
Steve Pomeranz: [LAUGH]
Annie Duke: [LAUGH] I mean so we don’t want to come to the conclusion that we’re a great drunk driver. And other things we’re doing, whether it’s like business or sales, or like the way we’re treating other people or our investment decisions.
Just because we have these good outcomes, to be like I’m a great drunk driver, we want to try to figure that out. So it’s good to change what your definition of being awesome is. So I like to think of awesome as doing the really hard work of, when I say, ooh, I had a really good result, but I think some of it was due to luck.
That actually makes me feel good. It’s taken a lot of training for me to get there. Or when I have a really bad result saying, you know what, I think I might have made a mistake here. That makes me feel good as well. And the reason why is that when I do that, what I’m trying to focus on is this is going to improve my decisions going forward.
So the next time I go to get in a car when maybe I’ve had too much to drink, I don’t do it.
Steve Pomeranz: Right.
Annie Duke: Because I’ve been willing to take responsibility.
Steve Pomeranz: Annie, let me move on here and talk about the necessity of the feedback. Now whenever you’re at the poker table and someone bets a certain amount in a certain way, that’s feedback.
You mentioned in the book this idea of conditioning and that you were using rats to test conditioning, and they would have a rat pull levers. And for every ten times the rat pressed the lever, the rat would get some food. But then in the experiment, they introduced some uncertainty where on average, every ten times the rat would get some food.
But it wasn’t exactly every ten times because he may press the lever 20 times and not get anything, or press it one time and get something. So now you’ve created amount of uncertainty even though the average amount of lever pulls is the same. In investing it’s very similar. I think saying the word average in investing is a very dangerous word.
And I often tell people, how can you drown in six inches of water when it’s average six inches. And the answer is well, it’s 50 feet in the middle of the lake. It’s centimeters at the beach part of the lake. So the average is six inches, but you can drown.
So average is dangerous. Take us into the world of average and uncertainty, and tell us what you make of that.
Annie Duke: So first of all, I really appreciate what you just said because I don’t think I’ve talked to anybody who’s talked about that, the bit about the classical conditioning and rats pressing levers that I have in the book.
So I really appreciate that. That’s the first, you are the first person who’s mentioned that, so thank you. [LAUGH]
Steve Pomeranz: That was not luck, by the way, that was skill. [LAUGH]
Annie Duke: Yes, no, that was totally skill. It was luck that you were the first person that mentioned it, though it was skill that you noticed the importance of it.
So the ideas that we can kind of think about what makes it easy to learn from something and what makes it hard to learn from something. So one of the things that make it easy, there’s two things that make things easy to learn from. The first is that the feedback comes in time for you to do something about it.
So obviously, if you started smoking in 1920, even though you’re destined for emphysema at some point, by the time that you get that feedback it’s too late to do anything about it. So we want to get something back that tells us about the quality of our decision quickly. That’s thing number one.
But also thing number two which is what you pointed out is that there is a really strong correlation between the feedback you get and the decisions that you’re making. And this is the difference between a rat pressing a lever every ten times which is very consistent feedback and introducing this uncertainty of on average.
Which makes it harder to know if I press the lever, am I going to get food or not, because it’s an on average thing. I don’t know if it’s going to be the next press or the next 20 presses, and that actually makes it really hard to learn from our outcomes.
Steve Pomeranz: Aren’t the slot machines based upon the fact that there may be a win on average every x amount of times, but the period is totally uncertain.
Annie Duke: Yes, and what’s interesting about that is that once you have people sort of trained on those slot machines, that they’re getting these variable rewards, you could actually turn the slot machine off in the sense of make it so it doesn’t pay out at all, and people would sit at those machines for a very long time and you know what they would say, right, I’, due.
Steve Pomeranz: I’m due. And it’s time, right? [LAUGH]
Annie Duke: Right, and so they’re not getting, they’re not learning from the feedback that they’re getting, that the machine isn’t paying out to actually get up and walk away. And I think that this is true in our lives as well is that we pick out, like we’re trying to find the signal and the noise, and we’re very often taking the wrong lesson.
So when we go back to this thing of self-serving bias, think about what that’s doing for you, right? It’s like if I have a good result, I’m going to repeat that decision over and over again. Well, why can you do that? Well, because the outcome could be because of your decisions, it could be because of luck, and we’re not 100% sure.
Steve Pomeranz: You’re never sure.
Annie Duke: So you choose to repeat the decision over and over again. It’s like continuing to press the lever. And now in the future when you don’t have the same good result that you had the first try, you pawn it off to luck.
Steve Pomeranz: Yeah.
Annie Duke: So how do you learn that? It’s like you have a good result, you’re like, I’m awesome, I’m going to keep doing that decision. And now you do that decision again, you have a bad result and you say, well, this time because of luck.
Steve Pomeranz: I think that happens a lot. I think that happens a lot in the investment world. I’ve been in it for over 30 years, and you see managers come and go. You see them with a particular strategy that seems to work extremely well. And you think it’s skill, but then the metrics of the world change.
And now that strategy stops working and you really wonder how much of that was skill, how much was luck. As a matter of fact, one of the biggest hedge funds just almost collapsed because they made a few terrible bets using the same methodology he was using that made him so successful in the first place.
So there’s this problem of, the problem is not the market, the problem is not that you’re not making enough money at work, the problem is you, that you haven’t really figured out how to overcome your biases.
Annie Duke: Yeah, and so I think that this is one of the lessons that is really important to take from this is that we’re not that different from rats in terms of the way that our brains are built when it comes to dealing with this kind of uncertainty, in the relationship between the feedback that we get and the actual quality of decisions. We just like to interpret the world to affirm our own beliefs to affirm own positive self-narrative and were kind of sort of stuck in that.
Steve Pomeranz: What do we do? What do we do to fix this?
Annie Duke: Well, yeah. So number one, let me just say like being smart isn’t particularly helpful. And if people are interested in that, they can go read the section in my book that’s about that. I think that it’s titled, “Being Smart Makes It Worse.” So just don’t think-
Steve Pomeranz: So what do you do if you’re smart? [LAUGH]
Annie Duke: Well-
Steve Pomeranz: You have to become humble, that’s what it is. You have to say the three words, I’m not sure.
Annie Duke: I’m not sure, and then understand the value of other people helping you in the process.
So I’m sure you’ve had this experience. Like you see somebody doing something and you say, or stating something or implying the same all strategy that you can see very clearly isn’t working anymore. And it’s so clear when somebody else is engaging in behavior that’s like biased or not rational, whatever, you can see it crystal clear.
But when it comes to your own behavior, it’s like when I talk about somebody else, like they won once and then they lost five times in a row and they’re just blaming it on luck. And you can say like, that’s really dumb. But you do it for yourself, you’re like no it really is luck this time because it’s me.
Steve Pomeranz: Right, okay so, number one
Annie Duke: So take that, get other people involved.
Steve Pomeranz: Okay, get other people involved. Try not to be too smart, right? I guess that’s it?
Annie Duke: Right.
Steve Pomeranz: I mean, even Buffett says that he’d rather hire someone who didn’t have a high IQ. It doesn’t take a high IQ to be successful in the world of investing. It takes you to understand your behavior and your biases and to learn from them.
Annie Duke: Yeah, so, A, start saying I’m not sure more and recognize that you need to be humble in the face of what you’re doing because what you’re doing is probably pretty complicated.
Because even driving a car is really complicated, and yet we’re all pretty overconfident about that. So number one, just recognize that what you’re doing is probably pretty hard. And while you might feel that you’re better on average at making those decisions than other people, you’re not better than the game that you’re playing.
So that’s number one. Number two is recognize the power of getting other people to help you. That it’s not shameful to say, I had a bunch of other people help work this decision with me. It’s not like you deserve less credit because you should get credit in the first place for recognizing that you’re better off bringing other viewpoints into the room.
That you’re better off having people challenge you about why you might be wrong. You’re better off, here is one of the key things. You’re better off having someone make you really rationally justify your decision, where you’re not allowed to just say, I know it, I feel it, this is my experience, I’m smart so I just know this is right.
But you have to actually be able to just defend your position. Because if you can’t defend your position to somebody who’s really honestly and seriously challenging you on it, then you don’t know anything about your decision. And that’s really sort of taken as part, if you can take John Stuart Mill to get to that.
And I think that that’s also a form of humility and also a form of recognizing that you don’t lose credit because you got help.
Steve Pomeranz: Okay, we have to stop there, the book, Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts. My guest Annie Duke studied cognitive science in school and is one of the top poker players in the United States.
Annie Duke: Well, I used to be, I retired in 2012.
Steve Pomeranz: Well, we never talked about the Celebrity Apprentice, and I’m so happy about that.
Annie Duke: I know.
Steve Pomeranz: You see how smart I am?
Annie Duke: Yeah.
Steve Pomeranz: Okay, anyway, Annie, thank you so much for joining us. And listen you all, if you have a question about what we just discussed, go to stevepomeranz.com and ask us anything you like. That’s stevepomeranz.com.
And while you’re there sign up for our weekly newsletter. It comes into your email. You can see all of our segments like this one. You can read it. You can listen to it, whatever you like, stevepomeranz.com. Annie, once again, thank you so much.
Annie Duke: Awesome, thank you for having me!
