<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Decisions in Context™]]></title><description><![CDATA[Decisions in Context™ explores how people, evidence, systems, and circumstances shape consequential decisions—and how Applied Decision Ecology can help improve them.]]></description><link>https://decisions.orblytics.com</link><image><url>https://substackcdn.com/image/fetch/$s_!sd9n!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78598693-26a2-49dc-96ec-574b697fe6d5_1254x1254.png</url><title>Decisions in Context™</title><link>https://decisions.orblytics.com</link></image><generator>Substack</generator><lastBuildDate>Mon, 16 Nov 2026 07:33:22 GMT</lastBuildDate><atom:link href="https://decisions.orblytics.com/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Matt Munyon]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[orblytics@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[orblytics@substack.com]]></itunes:email><itunes:name><![CDATA[Matt Munyon, PhD]]></itunes:name></itunes:owner><itunes:author><![CDATA[Matt Munyon, PhD]]></itunes:author><googleplay:owner><![CDATA[orblytics@substack.com]]></googleplay:owner><googleplay:email><![CDATA[orblytics@substack.com]]></googleplay:email><googleplay:author><![CDATA[Matt Munyon, PhD]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Should We Rely, Revisit, Refine, or Reject AI?]]></title><description><![CDATA[Decisions in Context&#8482; | A practical guide to deciding how much influence an AI answer deserves.]]></description><link>https://decisions.orblytics.com/p/should-we-trust-question-refine-or</link><guid isPermaLink="false">https://decisions.orblytics.com/p/should-we-trust-question-refine-or</guid><dc:creator><![CDATA[Matt Munyon, PhD]]></dc:creator><pubDate>Mon, 28 Sep 2026 22:10:46 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!sd9n!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F78598693-26a2-49dc-96ec-574b697fe6d5_1254x1254.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>&#8220;To trust AI&#8221; or &#8220;Not to trust AI&#8221;&#8212;that is the question leaders I work with nearly always ask me when considering AI adoption and use. And I nearly always tell them that their question is too broad to be useful.</p><p>I go on to explain that decisions, including ones about AI, need to be made in context. For example, AI can be helpful on one task and unreliable on another. Even within the same task, one AI output may be well supported while another may contain an error that reduces confidence in the answer. Trust, reduced to a yes-or-no decision, hides the part that I would argue matters most: whether this specific output deserves this specific level of influence over this specific decision.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://decisions.orblytics.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Decisions in Context&#8482;! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div><p>That is a question about calibrated reliance: adjusting how much we depend on an AI answer to the evidence for using it and the consequences of getting it wrong. A generally favorable opinion of AI does not settle that question. Neither does a generally unfavorable one.</p><h1><strong>Trust is too coarse a decision rule</strong></h1><p>The research gives us reasons to examine both responses. A review of 44 studies covering 122 tasks and 89,751 participants found algorithm aversion in 75% of the tasks (Kaufmann et al., 2023). People frequently favored human advice over algorithmic advice. That percentage describes the tasks reviewed; it is not the proportion of people who distrust AI, nor does it establish that every instance of reluctance was mistaken.</p><p>A broader review by Baines et al. (2024) helps explain why the pattern varies: advice-taking depends on the fit among the person, the AI advisor, and the decision environment. Some studies found people gave AI advice more weight than comparable human advice. In experiments by Cecil et al. (2024), incorrect AI advice impaired performance on a personnel-selection task despite the explanations provided. Klingbeil et al. (2024) also found that participants followed AI advice that conflicted with available contextual information.</p><p>Taken together, these findings make a blanket instruction to &#8220;trust AI more&#8221; difficult to defend. People can dismiss useful assistance or accept advice they should challenge. A Microsoft research synthesis describes appropriate reliance as accepting correct AI outputs and rejecting incorrect ones (Passi et al., 2024). The difficulty, of course, is knowing which is which before acting.</p><p>A fluent explanation does not resolve that difficulty on its own. It may make an answer easier to follow without giving us a way to check it. Fok and Weld (2024) argue that explanations support better AI-assisted decisions when they help people verify whether the recommendation is correct. I would therefore ask what an explanation lets me examine: a source, a calculation, an assumption, or a constraint that matters to the decision.</p><p>A recent preprint adds a useful qualification. Cohen et al. (2026) found that AI explanations supported the strongest performance on language-based logical reasoning tasks. For visual reasoning, probability displays&#8212;showing the model&#8217;s estimated likelihood for each answer&#8212;had the highest observed accuracy, but no additional support format significantly outperformed showing the predicted answer alone. That is a tentative result from a preprint, not an established rule for choosing an interface. It does reinforce the need to test support against the task rather than assume that more explanation will always help.</p><h1><strong>Four actions are better than one verdict</strong></h1><p>After leaders asked if they should trust AI, I soon learned that they were seeking a practical way to answer this question themselves and with their teams. &#8220;It depends&#8221; was not a sufficient answer. Drawing on this literature, I propose four responses to the output in front of us: rely, revisit, refine, or reject. I call this the <strong>4R AI Reliance Decision Framework</strong>. It is a practical framework informed by research; the framework itself has not yet been empirically tested.</p><p><strong>1. Rely. </strong>Choose this option when there is enough evidence that the output is suitable for its intended use. The task is clear, the relevant context is available, and the checks are proportionate to the consequences. You might rely on an AI summary of a short report after comparing its key claims with the report itself. That gives you a reason to use this summary; it does not establish that the system can now summarize anything reliably.</p><p><strong>2. Revisit. </strong>Re-examine the answer and its supporting evidence when it conflicts with other information, overlooks something important, or could lead to consequences you cannot readily undo. Ask what supports the recommendation and what might change it. Then examine the supporting material. Requesting sources is useful only if someone checks that they exist and support the claim. Another confident response from the same AI is not an independent check.</p><p><strong>3. Refine. </strong>Revise the task when the answer is potentially useful but the question, assumptions, or constraints were incomplete. Suppose an AI recommends expanding a service because demand is growing, but the prompt omitted staffing limits. Add that constraint and ask it to reassess the options. Refinement should improve the decision analysis, not keep prompting until the system agrees with a preferred answer. Check the revised output again.</p><p><strong>4. Reject. </strong>Set the output aside when a material error, fabricated source, or violation of an essential constraint makes it unsuitable for the decision. A recommendation that depends on a nonexistent study cannot support action, however plausible the surrounding prose. Return to reliable evidence or another method. Rejecting this answer does not require abandoning every useful application of AI.</p><p>These responses can overlap. You may question an assumption, refine the task, and then rely on part of the revised answer. You may also reject a recommendation while retaining a separately verified fact. What matters is being able to explain why the retained material deserves influence.</p><h1><strong>Make review possible in the actual workflow</strong></h1><p>Consider a hypothetical nonprofit using AI to recommend which service hours to reduce. The system identifies the evening shift as the least used. A manager checks the attendance totals and finds they are correct. Yet those totals do not reveal that some clients can attend only after work. The arithmetic survives review; the recommendation still needs questioning because attendance alone does not capture the consequences for access.</p><p>This is where Applied Decision Ecology changes the analysis. The output interacts with the organization&#8217;s goals, the information it collects, the people who review it, and the people affected. In this example, an efficiency measure could become a service rule unless someone brings the missing context into the decision. Improving the prompt may help, but the organization also needs a way for staff and clients to supply information the data leave out.</p><p>For that reason, I would define the review before putting AI into a consequential decision process. What can the reviewer check? Do they have the knowledge and time to do it? Can they stop or change the proposed action? Naming a human approver leaves those questions unanswered. If the review is expected to catch substantive errors, it needs resources and authority to match.</p><p>The checks should fit the consequences. A reversible wording choice may need a quick read. A service reduction calls for a closer examination of evidence, alternatives, and who bears the cost. When an answer cannot be adequately checked, that limitation should reduce its influence or lead us to defer the decision. Requiring a longer explanation will not necessarily supply the missing evidence.</p><p>There are limits to this approach. The studies discussed here cover particular tasks and settings; they do not prove that this four-action guide will improve every organizational decision. Human reviewers can also miss errors. I would test the process on a bounded task and record both useful assistance and failures: what was caught, what passed through, and how much checking the work required. Those observations can justify changing how much responsibility the system is given.</p><h1><strong>Better Decision Lesson</strong></h1><p>Before acting on an AI answer, decide what you need to check and what a mistake would cost. Use those answers to decide whether to rely, revisit, refine, or reject. The reason for your choice should be something you can explain to the people affected by the decision.</p><p>Where in your work is AI advice easiest to accept but hardest to check? That is a useful place to begin the conversation with your team.</p><h1><strong>References</strong></h1><p>Baines, J. I., Dalal, R. S., Ponce, L. P., &amp; Tsai, H.-C. (2024). Advice from artificial intelligence: A review and practical implications. <em>Frontiers in Psychology, 15</em>, 1390182. <a href="https://doi.org/10.3389/fpsyg.2024.1390182">https://doi.org/10.3389/fpsyg.2024.1390182</a></p><p>Cecil, J., Lermer, E., Hudecek, M. F. C., Sauer, J., &amp; Gaube, S. (2024). Explainability does not mitigate the negative impact of incorrect AI advice in a personnel selection task. <em>Scientific Reports, 14</em>, 9736. <a href="https://doi.org/10.1038/s41598-024-60220-5">https://doi.org/10.1038/s41598-024-60220-5</a></p><p>Cohen, R., Feng, L., Bloch, A., &amp; Kraus, S. (2026). <em>Supporting calibrated reliance in human-AI collaboration: Different strategies for different tasks</em> (Version 2) [Preprint]. arXiv. <a href="https://doi.org/10.48550/arXiv.2604.03237">https://doi.org/10.48550/arXiv.2604.03237</a></p><p>Fok, R., &amp; Weld, D. S. (2024). In search of verifiability: Explanations rarely enable complementary performance in AI-advised decision making. <em>AI Magazine, 45</em>(3), 317&#8211;332. <a href="https://doi.org/10.1002/aaai.12182">https://doi.org/10.1002/aaai.12182</a></p><p>Kaufmann, E., Chacon, A., Kausel, E. E., Herrera, N., &amp; Reyes, T. (2023). Task-specific algorithm advice acceptance: A review and directions for future research. <em>Data and Information Management, 7</em>(3), 100040. <a href="https://doi.org/10.1016/j.dim.2023.100040">https://doi.org/10.1016/j.dim.2023.100040</a></p><p>Klingbeil, A., Gr&#252;tzner, C., &amp; Schreck, P. (2024). Trust and reliance on AI&#8212;An experimental study on the extent and costs of overreliance on AI. <em>Computers in Human Behavior, 160</em>, 108352. <a href="https://doi.org/10.1016/j.chb.2024.108352">https://doi.org/10.1016/j.chb.2024.108352</a></p><p>Passi, S., Dhanorkar, S., &amp; Vorvoreanu, M. (2024). <em>Appropriate reliance on generative AI: Research synthesis</em> (MSR-TR-2024-7). Microsoft Research. <a href="https://www.microsoft.com/en-us/research/publication/appropriate-reliance-on-generative-ai-research-synthesis/">https://www.microsoft.com/en-us/research/publication/appropriate-reliance-on-generative-ai-research-synthesis/</a></p><h1>About the Author</h1><p>Matthew D. Munyon, Ph.D., is the founder of Orblytics LLC and serves as Chief Decision Scientist and Chief Peace Officer. His work brings together decision science, behavioral science, data science, AI, and systems thinking to help people and organizations make better decisions. Through Applied Decision Ecology he examines how evidence, people, and context shape decisions and their consequences. Through Applied Decision Ecology, he examines how evidence, people, and context shape decisions and their consequences. Applied Decision Ecology is an interdisciplinary framework he is developing that integrates decision science, behavioral science, and systems thinking to inform real-world decisions. A central idea in his work is that better decisions can create greater peace by reducing avoidable uncertainty, conflict, friction, and disorder&#8212;and that greater peace, in turn, can create conditions for clearer judgment and better decisions. He views this reciprocal relationship as a reinforcing cycle supporting both performance and well-being.</p><div class="subscription-widget-wrap-editor" data-attrs="{&quot;url&quot;:&quot;https://decisions.orblytics.com/subscribe?&quot;,&quot;text&quot;:&quot;Subscribe&quot;,&quot;language&quot;:&quot;en&quot;}" data-component-name="SubscribeWidgetToDOM"><div class="subscription-widget show-subscribe"><div class="preamble"><p class="cta-caption">Thanks for reading Decisions in Context&#8482;! Subscribe for free to receive new posts and support my work.</p></div><form class="subscription-widget-subscribe"><input type="email" class="email-input" name="email" placeholder="Type your email&#8230;" tabindex="-1"><input type="submit" class="button primary" value="Subscribe"><div class="fake-input-wrapper"><div class="fake-input"></div><div class="fake-button"></div></div></form></div></div>]]></content:encoded></item></channel></rss>