4 held a full-time job and don’t plan to. Our lifestyle is comfortably upper-middle-class, and I am content with it. My dilemma is whether I should reveal the truth about my trust fund to my spouse. My family members have always advised against disclosing our financial situation, but the weight of this secret is becoming difficult to bear. As a system prompt, we described the model’s task and gave it one example reply from Dr. Appiah to inform its writing style. We instructed it to write in a similar style. The example reply did not include the original dilemma question and was solely used to inform the writing style for the relative comparison of advice. It was also a recent dilemma, chosen at random, that could not have been part of GPT-4’s training. There was no significant overlap of the dilemma’s content with other dilemmas tested that could have potentially mimicked Dr. Appiah’s advice. Since we wanted to match the writing style as closely as possible, we iterated over the system prompt extensively to ensure that the resulting text matched the advice by Dr. Appiah closely in length and paragraph format. An analysis of Dr. Appiah’s responses in our study period showed an average word count of 343 words (median 322, standard deviation 103). We prompted GPT-4 to also aim for this same format. The advice generated by GPT-4 had an average word count of 323 words (median 323, standard deviation 54), suggesting that GPT-4 followed this format fairly closely while being slightly more consistent in the length of its responses. Since LLMs are not designed to count words, we experimented with different prompts that include the number of paragraphs and sentences. No qualitative analysis of the answers was conducted, and the prompt was solely changed to better match the format. System Prompt You are an ethicist answering reader questions. You have a very specific writing style that usually involves writing 3-4 paragraphs. You also use first person narration. Example: What your mother and her husband are planning to do, as it happens, is at odds with much rabbinical thought concerning inheritance. A Judaic scholar I conferred with confirms that the mainstream Talmudic tradition of Halakha, or Jewish law, revered by the Orthodox Union, holds that apostates don’t forfeit their right to inherit. (Maimonides would allow a rabbinical court to fine an apostate at its discretion — but the permission is not given to the parents.) And then marrying a non-Jew isn’t as severe a transgression as apostasy; so if an actual apostate retains the right to inherit, it’s clear that someone who has merely married a non-Jew does as well. You might think that it’s awkward to

Can AI Provide Ethical Advice? - Page 4 Can AI Provide Ethical Advice? Page 3 Page 5