Featured Post from Emory University, Goizueta Business School

AI-Generated Content is a Game Changer for Marketers, but at What Cost?

Feb 6, 2023

6 min

David Schweidel
Goizueta’s David Schweidel pitted man against the machine to create SEO web content only to find that providing an editor with bot-generated content trounces the human copywriter every time. Good news for companies looking to boost productivity and save cash, he says. But could there be other hidden costs?




In December 2022, The New York Times ran a piece looking back on the year’s biggest consumer tech updates. The review was mixed. Ownership shifts in the world of social media garnered special mentions, but hardware innovations had been largely “meh,’ mused the Times. There was one breakthrough area that warranted attention, however: AI-powered language-processing tech capable of generating natural-looking text, the same technology that powers familiar chatbots. And one such technology could well be poised to “invade our lives in 2023.”


Earlier in December, AI research lab OpenAI, released the latest update to its Generative Pre-Trained Transformer technology, an open source artificial intelligence. It’s latest iteration, ChatGPT, immediately went viral. Here was an AI assistant that sounded intelligent. Not only could it answer any question thrown its way without supervised training, but when prompted, it could also write blog posts, as well as find and fix bugs in programming code. ChatGPT could draft business proposals and even tell jokes. All of this at a speed that beggared belief.


Since its first release in 2020, OpenAI’s GPT technology has powered through a slew of updates that have seen its capabilities leap forward “by light years” in less than 24 months, says Goizueta Professor of Marketing, David Schweidel. For businesses looking to harness this rapidly-evolving technology, the potential is clearly enormous. But aren’t there also risks that industry and consumers alike will need to navigate?



Schweidel is clear that the academic community and initiatives such as the Emory AI Humanity Initiative have a critical role in asking hard questions—and in determining the limitations and dangers, as well as the opportunities, inherent in tech innovation—because, as he puts it, “these things are going to happen whether we like it or not.”


Man Versus Machine


To that end, Schweidel and colleagues from Vienna University of Economics and Business and the Modul University of Vienna have put together a study looking at how well natural language generation technologies perform in one specific area of marketing: drafting bespoke content for website search engine optimization, better known as SEO. What they find is that content crafted by the machine, after light human editing, systematically outperforms its human counterparts—and by a staggering margin. Digging through the results, Schweidel and his colleagues can actually pinpoint an almost 80 percent success rate for appearing on the first page of search engine results with AI-generated content. This compares with just 22 perfect of content created by human SEO experts. In other words, the AI content passed to a human is roughly four times more effective than a skilled copywriter working alone.


Reaching these findings meant running two real-time, real-world experiments, says Schweidel. First, he and his colleagues had to program the machine, in this case GPT 2, an earlier incarnation of GPT. GPT relies on natural language generation (NGL), a software process that converts manually uploaded input into authentic-sounding text or content—comparable in some ways to the human process of translating ideas into speech or writing. To prepare GPT-2 for SEO-specific content creation, Schweidel et al. started with the pre-trained GPT-2, and then let the machine do the heavy lifting: searching the internet for appropriate results based on the desired keyword, scraping the text of the websites, and updating GPT-2 to “learn” what SEO looks like, says Schweidel.


We partnered with an IT firm and a university to run our field experiments. This meant creating SEO content for their websites using GPT-2 and actual human SEO experts, and then doing A/B testing to see which content was more successful in terms of landing in the top 10 search engine results on Google. So this was an opportunity to put the AI bot to the test in a real-world setting to see how it would perform against people.


The results point to one clear winner. Not only did content from GPT-2 outperform its human rivals in SEO capabilities, it did so at scale. The AI-generated content scored a daily median result of seven or more hits in the first page of Google search results. The human-written copy didn’t make it onto the first result page at all. On its best day, GPT showed up for 15 of its 19 pages of search terms inside the top 10 search engine results page, compared with just two of the nine pages created by the human copywriters—a success rate of just under 80 percent compared to 22 percent.


Savings at Scale


The machine-generated content, after being edited by a human, trounces the human in SEO. But that’s not all, says Schweidel. The GPT bot was also able to produce content in a fraction of the time taken by the writers, reducing production time and associated labor costs by more than 90 percent, he says.


“In our experiments, the copywriters took around four hours to write a page, while the GPT bot and human editor took 30 minutes. Now assuming the average copywriter makes an annual $45K on the basis of 1,567 hours of work, we calculate that the company we partnered with would stand to save more than $100,000 over a five-year period just by using the AI bot in conjunction with a human editor, rather than relying on SEO experts to craft content. That’s a 91 percent drop in the average cost of creating SEO content. It’s an orders of magnitude difference in productivity and costs.”


But there are caveats.


First off, there’s the quality of the machine-generated content to consider. For all its mind-boggling capabilities, even the newly released ChatGPT tends to read somewhat sterile, says Schweidel. That’s a problem both in terms of Google guidelines and brand coherence. Human editors are still needed in order to attenuate copy that can sound a little “mechanical.”


“Google is pretty clear in its guidelines: Content generated by machines alone is a definite no-no. You also need to factor in the uncanny valley effect whereby something not quite human can come off as weird. Having an editor come in to smooth out AI content is critical to brand voice as well as the human touch.”



Asking the Big Questions


Then there are the moral and metaphysical dimensions of machine learning and creativity that beg an important question: Just because we can, does that mean we should? Here, Schweidel has grave reservations about the future of ChatGPT and its ilk.


The potential of this kind of technology is extraordinarily exciting when you think about the challenges we face from productivity to pandemics, from sustainable growth to climate change. But let’s be very clear about the risks, too. AI is already capable of creating content—audio, visual and written—that looks and feels authentic. In a world that is hugely polarized, you have to ask yourself: How can that be weaponized?


At the end of the day, says Schweidel, the large language models powering these generative AIs are essentially “stochastic parrots:” trained mimics whose output can be hard to predict. In the wrong hands, he warns, the potential for misinformation—and worse—could well be “terrifying.”


“Shiny new tech is neither inherently good nor bad. It’s human nature to push the boundaries. But we need to ensure that the guardrails are in place to regulate innovation at this kind of pace, and that’s not easy. Governments typically lag far behind OpenAI and companies like them, even academics have a hard time keeping up. The real challenge ahead of us will be about innovating the guardrails in tandem with the tech—innovating our responsible practices and processes. Without effective safeguards in place, we’re on a path to potential destruction.”


Covering AI or interesting in knowing more about this fascinating topic - then let our experts help with your coverage and stories.


David Schweidel is the Rebecca Cheney McGreevy Endowed Chair and Professor of Marketing at Emory University's Goizueta Business School. Simply click on David's icon now to arrange an interview today.

Connect with:
David Schweidel

David Schweidel

Professor of Marketing & Goizueta Chair in Business Technology

Marketing analytics expert focused on the opportunities at the intersection of marketing and technology

Marketing TechnologyAISocial MediaPolitical MarketingCustomer Analytics

You might also like...

Check out some other posts from Emory University, Goizueta Business School

2 min

Goizueta Professors on Olympic Fandom, Viewership, Economics of the Games

As we countdown to the July 26 opening of the Olympic Games in Paris, we have two Goizueta Business School experts who can discuss the Olympics, including fandom, viewership, and the economics of the Paris Games. Professor Mike Lewis studies fandom and has spent many years tracking how sports can stay relevant and lucrative across generations. “While the Olympics is presented as a pure sports competition, it’s actually a unique sports entertainment event that leverages underpaid athletes and national identities to create one of the foremost sports-based advertising platforms.” His research on this year’s Olympics finds: Viewership skews to an older audience - with Baby Boomers showing the biggest fandom. The Games have an ongoing challenge to attract a younger audience. The Olympic broadcast will have to work hard to gain the attention of Gen Z and Millennials. Olympic fans are pretty even between genders, which is very different from typical sports fans that are more than 50% male. Check out his latest research: And his latest podcast on the Games: Professor Tom Smith studies the economics of sports. He can discuss: The economic impact of the Games for Paris during what is usually a high tourist season. The economics of decisions by city leaders to repurpose current facilities, landscape and architecture for the Paris games. What other cities can learn from the Paris games. Both Tom and Mike are available to speak with media about the upcoming Olympic Games - simply click on either expert's icon now to arrange an interview today.

10 min

Expert Insight: Training Innovative AI to Provide Expert Guidance on Prescription Medications

A new wave of medications meant to treat Type II diabetes is grabbing headlines around the world for their ability to help people lose a significant amount of weight. They are called GLP-1 receptor agonists. By mimicking a glucagon-like peptide (GLP) naturally released by the body during digestion, they not only lower blood sugar but also slow digestion and increase the sense of fullness after eating. The two big names in GLP-1 agonists are Ozempic and Wegovy, and both are a form of semaglutide. Another medication, tirzepatide, is sold as Mounjaro and Zepbound. It is also a glucose-dependent insulinotropic polypeptide (GIP) agonist as well as GLP-1. Physicians have been prescribing semaglutide and tirzepatide with increasing frequency. However, both medications come with a host of side effects, including nausea and stomach pain, and are not suitable for every patient. Many clinics and physicians do not have immediate access to expert second opinions, as do the physicians at Emory Healthcare. Creating a Digital Twin That lack of an expert is one of the reasons Karl Kuhnert, professor in the practice of organization and management at Emory University’s Goizueta Business School, is using artificial intelligence to capture the expertise of physicians like Caroline Collins MD through the Tacit Object Modeler™, or TOM. By using TOM, developed by Merlynn Intelligence Technologies, Kuhnert and Collins can create her “decision-making digital twin.” This allows Collins to reveal her expertise as a primary care physician with Emory Healthcare and an Assistant Professor at Emory School of Medicine, where she has been leading the field in integrating lifestyle medicine into clinical practices and education. Traditional AI, like ChatGPT, uses massive amount of data points to predict outcomes using what’s known as explicit knowledge. But it isn’t necessarily learning as it goes. According to Kuhnert, TOM has been designed to learn how an expert, like Collins, decides whether or not to prescribe a drug like semaglutide to a patient. Wisdom or tacit knowledge is intuitive and rooted in experience and context. It is hard to communicate, and usually resides only in the expert’s mind. TOM’s ability to “peek into the expert’s mind makes it a compelling technology for accessing wisdom.” “Objective or explicit knowledge is known and can be shared with others,” says Kuhnert. "For example, ChatGPT uses explicit knowledge in its answers. It’s not creating something new. It may be new to you as you read it, but somebody, somewhere, before you, has created it. It’s understood as coming from some source." Karl Kuhnert “Tacit knowledge is subjective wisdom. Experts offer this, and we use their tacit know-how, their implicit knowledge, to make their decisions. If it were objective, everyone could do it. This is why we hire experts: They see things and know things others don’t; they see around corners.” Mimicking the Mind of a Medical Expert Teaching TOM to see around the corners requires Collins to work with the AI over the course of a few days. “Essentially what I do is I sit down with, in this case, a physician, and ask them, ‘What are thinking about when you make this decision?'” says Kuhnert. “The layperson might think that there are hundreds of variables in making a medical decision like this. With the expert’s tacit knowledge and experience, it is usually between seven and twelve variables. They decide based on these critical variables,” he says. "These experts have so much experience, they can cut away a lot of the noise around a decision and get right to the point and ask, ‘What am I looking at?’" Karl Kuhnert As TOM learns, it presents Collins with more and different scenarios for prescribing semaglutide. As she makes decisions, it remembers the variables present during her decision-making process. “Obviously, some variables are going to be more important than other variables. Certain combinations are going to be challenging,” says Collins. “Sometimes there are going to be some variables where I think, yes, this patient needs a GLP-1. Then there may be some variables where I think, no, this person really doesn’t need that. And which ones are going to win out? That’s really where TOM is valuable. It can say, okay, when in these difficult circumstances where there are conflicting variables, which one will ultimately be most important in making that decision?” The Process: Trusting AI After working with TOM for several hours, Collins will have reacted to enough scenarios for TOM to learn to make her decision. The Twin will need to demonstrate that it can replicate her decision-making with acceptable accuracy—high 90s to 100 percent. Once there, Collins’ Twin is ready to use. “I think it’s important to have concordance between what I would say in a situation and then what my digital twin would say in a situation because that’s our ultimate goal is to have an AI algorithm that can duplicate what my recommendation would be given these circumstances for a patient,” Collins says. “So, someone, whether that be an insurance company, or a patient themselves or another provider, would be able to consult TOM, and in essence, me, and say, in this scenario, would you prescribe a GLP-1 or not given this specific patient’s situation?” The patient’s current health and family history are critical when deciding whether or not to prescribe semaglutide. For example, according to Novo Nordisk, the makers of Ozempic, the drug should not be prescribed to patients with a history of problems with the pancreas or kidneys or with a family history of thyroid cancer. Those are just the start of a list of reasons why a patient may or may not be a good candidate for the medication. Kuhnert says, “What we’re learning is that there are so many primary care physicians right now that if you come in with a BMI over 25 and are prediabetic, you’re going to get (a prescription). But there’s much more data around this to suggest that there are people who are health marginalized, and they can’t do this. They should not have this (medication). It’s got to be distributed to people who can tolerate it and are safe.” Accessing the Digital Twin on TOM Collins’s digital twin could be available via something as easy to access as an iPhone app. “Part of my job is to provide the latest information to primary care physicians. Now, I can do this in a way that is very powerful for primary care physicians to go on their phones and put it in. It’s pretty remarkable, according to Colllins.” It is also transparent and importantly sourced information. Any physician using a digital twin created with TOM will know exactly whose expertise they are accessing, so anyone asking for a second opinion from Colllins will know they are using an expert physician from Emory University. In addition to patient safety, there are a number of ways TOM can be useful to the healthcare industry when prescribing medications like semaglutide. This includes interfacing with insurance companies and the prior approval process, often lengthy and handled by non-physician staff. “Why is a non-expert at an insurance company determining whether a patient needs a medication or not? Would it be better to have an expert?” says Collins. “I’m an expert in internal medicine and lifestyle medicine. So, I help people not only lose weight, but also help people change their behaviors to optimize their health. My take on GLP-1 medications is not that everyone needs them, it’s that they need to be utilized in a meaningful way, so patients will get benefit, given risks and benefits for these medications.” The Power of a Second Opinion Getting second, and sometimes third, opinions is a common practice among physicians and patients both. When a patient presents symptoms to their primary care physician, that physician may have studied the possible disease in school but isn’t necessarily an expert. In a community like Emory Healthcare, the experts are readily available, like Collins. She often serves as a second opinion for her colleagues and others around the country. “What we’re providing folks is more of a second opinion. Because we want this actually to work alongside someone, you can look at this opinion that this expert gave, and now, based on sourced information, you can choose. This person may be one of the best in the country, if not the world, in making this decision. But we’re not replacing people here. We’re not dislocating people with this technology. We need people. We need today’s and tomorrow’s experts as well,” according to Kuhnert. But also, you now have the ability to take an Emory physician’s diagnosing capabilities to physicians in rural areas and make use of this information, this knowledge, this decision, and how they make this decision. We have people here that could really help these small hospitals across the country. Caroline Collin MD Rural Americans have significant health disparities when compared to those living in urban centers. They are more likely to die from heart disease, cancer, injury, chronic respiratory disease, and stroke. Rural areas are finding primary care physicians in short supply, and patients in rural areas are 64 percent less likely to have access to medical specialists for needed referrals. Smaller communities might not have immediate access to experts like a rheumatologist, for example. In addition, patients in more rural areas might not have the means of transportation to get to a specialist, nor have the financial means to pay for specialized visits for a diagnosis. Collins posits that internal medicine generalists might suspect a diagnosis but want to confirm before prescribing a course of treatment. “If I have a patient for whom I am trying to answer a specific question, ‘Does this patient have lupus?’, for instance. I’m not going to be able to diagnose this person with lupus. I can suspect it, but I’m going to ask a rheumatologist. Let’s say I’m in a community where unfortunately, we don’t have a rheumatologist. The patient can’t see a rheumatologist. That’s a real scenario that’s happening in the United States right now. But now I can ask the digital twin acting as a rheumatologist, given these variables, ‘Does this patient have lupus?’ And the digital twin could give me a second opinion.” Sometimes, those experts are incredibly busy and might not have the physical availability for a full consult. In this case, someone could use TOM to create the digital twin of that expert. This allows them to give advice and second opinions to a wider range of fellow physicians. As Kuhnert says, TOM is not designed or intended to be a substitute for a physician. It should only work alongside one. Collins agreed, saying, “This doesn’t take the place of a provider in actual clinical decision-making. That’s where I think someone could use it inappropriately and could get patients into trouble. You still have to have a person there with clinical decision-making capacity to take on additional variables that TOM can’t yet do. And so that’s why it’s a second opinion.” “We’re not there yet in AI says Collins. We have to be really careful about having AI make actual medical decisions for people without someone there to say, ‘Wait a minute, does this make sense?’” AI Implications in the Classroom and Beyond Because organizations use TOM to create digital twins of their experts, the public cannot use the twins to shop for willing doctors. “We don’t want gaming the system,” says Collins. “We don’t want doctor shopping. What we want is a person there who can utilize AI in a meaningful way – not in a dangerous way. I think we’ll eventually get there where we can have AI making clinical decisions. But I don’t think I’d feel comfortable with that yet.” The implications of using decision-making digital twins in healthcare reach far beyond a second opinion for prescription drugs. Kuhnert sees it as an integral part of the future of medical school classrooms at Emory. In the past, teaching case studies have come from books, journals, and papers. Now, they could come alive in the classroom with AI simulation programs like TOM. "I think this would be great for teaching residents. Imagine that we could create a simulation and put this in a classroom, have (the students) do the simulation, and then have the physician come in and talk about how she makes her decisions." Karl Kuhnert “And then these residents could take this decision, and now it’s theirs. They can keep it with them. It would be awesome to have a library of critical health decisions made in Emory hospitals,” Kuhnert says. Collins agreed. “We do a lot of case teaching in the medical school. I teach both residents and medical students at Emory School of Medicine. This would be a really great tool to say, okay, given these set of circumstances, what decision would you make for this patient? Then, you could see what the expert’s decision would have been. That could be a great way to see if you are actually in lockstep with the decision-making process that you’re supposed to be learning.” Kuhnert sees decision-making twins moving beyond the healthcare system and into other arenas like the courtroom, public safety, and financial industries and has been working with other experts to digitize their knowledge in those fields. "The way to think about this is: say there is a subjective decision that gets made that has significant ramifications for that company and maybe for the community. What would it mean if I could digitize experts and make it available to other people who need an expert or an expert’s decision-making?" Karl Kuhnert “You think about how many people aren’t available. Maybe you have a physician who’s not available. You have executives who are not available. Often expertise resides in the minds of just a few people in an organization,” says Kuhnert. “Pursuing the use of technologies like TOM takes the concept of the digital human expert from simple task automation to subjective human decision-making support and will expand the idea of a digital expert into something beyond our current capabilities,” Kuhnert says. “I wanted to show that we could digitize very subjective decisions in such areas as ethical and clinical decision-making. In the near future, we will all learn from the wisdom codified in decision-making digital twins. Why not learn from the best? There is a lot of good work to do.” Karl Kuhnert is a Professor in the Practice of Organization & Management and Associate Professor of Psychiatry, School of Medicine and Senior Faculty Fellow of the Emory Ethics Center. If you're looking to connect with Karl to know more - simply click on his icon now to arrange a time to talk today.

4 min

Decoding Hierarchies in Business: When is Having a Boss a Benefit for an Organization?

Most companies around the world have a leader, whether that title is a President, CEO, or Founder. There’s almost always someone at the very top of a corporate food chain, and from that position down, the company is structured hierarchically, with multiple levels of leadership supervising other employees. It’s a structure with which most people in the working world are familiar, and it dates back as long as one can remember. The word itself—leader—dates back to as far as the 12th Century and is derived from the Old English word “laedere,” or one who leads. But in 2001, a group of software engineers developed the Agile Workflow Methodology, a project development process that puts a priority on egalitarian teamwork and individual independence in searching for solutions. A number of businesses are trying to embrace a flatter internal structure, like the agile workflow. But is it necessarily the best way to develop business processes? That’s the question posed by researchers, including Goizueta Business School’s Özgecan Koçak, associate professor of organization and management, and fellow researchers Daniel A. Levinthal and Phanish Puranam in their recently published paper on organizational hierarchies. “Realistically, we don’t see a lot of non-hierarchical organizations,” says Koçak. “But there is actually a big push to have less hierarchy in organizations.” "Part of it is due to the demotivating effects of working in authoritarian workplaces. People don’t necessarily like to have a boss. We place value in being more egalitarian, more participatory." Özgecan Koçak, Associate Professor of Organization & Management “So there is some push to try and design organizations with flatter hierarchies. That is specifically so in the context of knowledge-based work, and especially in the context of discovery and search.” Decoding Organizational Dynamics While the idea of an egalitarian workplace is attractive to many people, Koçak and her colleagues wanted to know if, or when, hierarchies were actually beneficial to the health of organizations. They developed a computational agent-based model, or simulation, to explore the relationships between structures of influence and organizational adaptation. The groups in the simulation mimicked real business team structures and consisted of two types of teams. In the first type, one agent had influence over the beliefs of rest of the team. For the second type, no one individual had any influence over the beliefs of the team. The hierarchical team vs. the flat structured team. “When you do simulations, you want to make sure that your findings are robust to those kinds of things like the scale of the group, or the how fast the agents are learning and so forth,” says Koçak. "What’s innovative about this particular simulation is that all the agents are learning from their environment. They are learning through trial and error. They are trying out different alternatives and finding out their value." Özgecan Koçak Koçak is very clear that the hierarchies in the simulation are not exactly like hierarchies in a business organization. Every agent was purposefully made to be the same without any difference in wisdom or knowledge. “It’s really nothing like the kinds of hierarchies you would see in organizations where there is somebody who has a corner office, or somebody who is has a management title, or somebody’s making more than the others. In the simulation, it’s nothing to do with those distributional aspects or control, and nobody has the ability to control what others do in (the simulation). All control comes through influence of beliefs.” Speed vs. Optimal Solutions What they found in the simulation was that while both teams solved the same problems presented to them, they achieved different results at different speeds. "We find that hierarchical teams don’t necessarily find the best solution, but they find the good enough solution in the shorter term. So if you are looking at the really long term, crowds do better. The crowds where individuals are all learning separately, they find the best solution in the long run, even though they are not learning from each other." Özgecan Koçak For example, teams of scientists looking for cures or innovative treatments for diseases work best with a flat structure. Each individual works on their own timeline, with their own search methodologies. The team only comes together for status updates or to discuss their projects without necessarily getting influence or direction from colleagues. The long-term success of the result is more important in some cases than the speed at which they arrive to their conclusion. That won’t work for an organization that answers to a board of directors or shareholders. Such parties want to see rapid results that will quickly impact the bottom line of the company. This is why the agile methodology is not beneficial to large-scale corporations. Koçak says, “When you try to think about an entire organization, not just teams, it gets more complicated. If you have many people in an organization, you can’t have everybody just be on the same team. And then you have to worry about how to coordinate the efforts of multiple teams. "That’s the big question for scaling up agile. We know that the agile methodology works pretty well at the team level. However, when firms try to scale it up applied to the entire organization, then you have more coordination problems." Özgecan Koçak Özgecan Koçak (pronounced as ohz-gay-john ko-chuck) is associate professor of Organization & Management at Emory University’s Goizueta Business School. If you're looking to know more about this topic or connect with Özgecan for an interview - simply click on her icon today

View all posts
Powered by