ChatGPT responses are as good as your data and prompt

ChatGPT, System Role Property, Data Quality

ChatGPT responses are as good as your data and prompt

Brad Crain, 10 min read

Like many others, I've discovered that ChatGPT's accuracy isn't always perfect. This realization can be concerning, especially when deploying it for use cases where precision matters. Companies prioritize the integrity and accuracy of their communications with clients and prospects, and rightly so. So, when relying on ChatGPT—whether directly or via its API—for information retrieval and composing responses, attention to accuracy is crucial.

Real-World Example: Assisting Older Adults with Service Providers

Recently, I was developing a chatbot to help older adults find service providers across different regions. Initially, I provided ChatGPT with a list of providers in the form of a GitHub Markup Table. However, when I asked ChatGPT to determine the total number of providers covering a specific region, the response provided an incorrect count. Naturally, I double-checked my table data to ensure proper formatting.

To improve accuracy, I made two key adjustments:

Remember, ChatGPT's responses are only as good as the data you provide and the prompts you use. With thoughtful adjustments, you can enhance its performance and reliability.

(content enhanced using Microsoft CoPilot)

**

if you are curious, here is an example of what I did

Data passed to Chat Completions API

[

  {

    "role": "system",

    "content": "You are a data analyst. You provide information about the PIP directory. Your responses are based exclusively on information you find in the business profile information or calculations you compute using this information.  The business profile information is provided to you in the system role context and is delimited by triple quotes. Instructions: 1) Review the information in the data table. You can ignore spaces and tab characters. 2) Read and process the information provided.  When developing your response, it is required that you check each row methodically against the criteria before providing your answer. Use step by step reasoning in developing your responses. 3) Always double check your answers and make any corrections to errors discovered during this check process before responding to the user.  You do not inform the users about your process of double checking your answers. This is important.  4) in your reply never mention that you get information from the system role context property, this is very important, 5) If you think your response may be inaccurate or vague, do not write it and answer with the exact text \"I don't have a response.\", 6) your response  is always fully complete and never shortened or made brief 7) you do not ask the user if they have any more questions. this is important.\"\"\"Business profile information is as follows Company name:Island Senior Resources PIP Directory. Company website:https://senior-resources.org/pip. Company email:reception@islandseniorservices.org. Company phone:360-321-1600. Company description:Service:  Private In-Home Help Directory\r\nDescription: Island Senior Resource s Private In-Home Provider (PIP) directory for Whidbey Island   previously referred to as the PIP list   might have just the person you need. Each provider has passed a background check. As of August 2023, the Private In-Home Provider Directory will be updated twice a year. The next Private In-Home Provider List update will take place in January 2024.\r\n\r\nNote - It is your responsibility to interview, check references, and negotiate duties and pay. See our How to Hire Guide for hiring tips and information regarding certifications or licenses. Island Senior Resources maintains this directory of providers as a resource for the community. This is only a list of people who are interested in working for seniors. This directory is not a recommendation. The providers listed have no affiliation with Island Senior Resources, therefore this agency is not responsible for their actions.. Product description:Here is a list of PIP providers. It is in the CSV format.  The list is:\r\nUnique_Provider_number,Provider_name,Provides_services_to_Camano_Island_residents_(Yes/No),Provides_services_to_North_Whidbey_Island_residents_(Yes/No),Provides_services_to_Central_Whidbey_Island_residents_(Yes/No),Provides_services_to_South_Whidbey_Island_residents_(Yes/No),Phone,Provides_service_type_-_Personal_care_(Yes/No),Provides_service_type_-_Transfers_(Yes/No),Provides_service_type_-_Homemaking_(Yes/No),Provides_service_type_-_Chores_(Yes/No),Provides_service_type_-_Respite_care_(Yes/No),Provides_service_type_-_Companionship_(Yes/No),Provides_service_type_-_Transport_(Yes/No),Provides_service_type_-_Engagement_(Yes/No),Provides_service_type_-_Overnight_(Yes/No),Additional_information_about_the_provider,If_Provides_Additional_Services_describe_it_or_enter_No\r\n1,Brad CNotWhidbey,No,No,No,No,425-445-0767,No,No,No,No,No,No,No,No,No,filler filler,No \r\n2,Aubrie LNotWhidbey,No,No,No,No,425-220-5016,No,No,No,No,No,No,No,No,No,filler filler,No \r\n3,Azrael R,No,No,No,Yes,360-914-2268,Yes,Yes,Yes,No,Yes,Yes,Yes,No,Yes,filler filler,No \r\n4,Barbara D,No,No,No,Yes,425-516-9183,Yes,No,Yes,No,No,No,No,No,No,filler filler,No \r\n5,Barbara M,No,No,No,Yes,360-967-6047,No,No,Yes,Yes,Yes,Yes,Yes,Yes,No,filler filler,No \r\n6,Bernadete D,No,Yes,Yes,Yes,360-730-1652,Yes,Yes,Yes,No,Yes,Yes,No,No,Yes,filler filler,No \r\n7,Briar R,No,No,No,Yes,360-914-2270,No,No,No,No,No,Yes,No,No,No,filler filler,No \r\n8,Caroline F,No,Yes,Yes,Yes,757-812-2778,Yes,Yes,Yes,No,Yes,Yes,No,No,Yes,filler filler,No \r\n9,Cathleen I,Yes,No,Yes,Yes,307-690-8709,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,filler filler,yardwork \r\n10,Charlton W,No,Yes,Yes,Yes,360-682-8791,Yes,No,Yes,Yes,No,Yes,Yes,Yes,Yes,filler filler,handywork painting yardwork \r\n11,Christine L,Yes,Yes,Yes,Yes,206-946-2094,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,Yes,filler filler,handywork yardwork \r\n12,Cindy H,No,No,No,Yes,360-391-1378,Yes,No,Yes,No,Yes,Yes,Yes,No,No,filler filler,No \r\n13,Colleen M,No,No,No,Yes,360-296-5065,Yes,Yes,Yes,Yes Yes,Yes,Yes,Yes,Yes,filler filler,No \r\n14,Crystel M,No,Yes,Yes,Yes,360-989-0119,Yes,No,Yes,Yes,Yes,Yes,Yes,Yes,No,filler filler,handywork painting yardwork hauling \r\n15,Darcy L,No,No,No,Yes,360-320-5391,Yes,No,Yes,Yes,No,Yes,No,Yes,No,filler filler,No \r\n16,Debbie N,No,No,No,Yes,360-612-7314,Yes,Yes,Yes,No,Yes,Yes,Yes,No,No,filler filler,No \r\n17,Deborah S,No,No,Yes,Yes,360-708-3568,Yes,Yes,Yes,No,Yes,Yes,Yes,Yes,No,filler filler,No \r\n18,Debra W,Yes,Yes,Yes,Yes,360-720-9942,Yes,Yes,Yes,Yes,No,Yes,Yes,Yes,Yes,filler filler,yardwork \r\n19,Debrax B,No,No,No,Yes,360-824-1026,No,No,Yes,No,No,Yes,No,No,No,filler filler,No \r\n20,Dominique K,No,No,No,Yes,206-962-9354,Yes,Yes,Yes,Yes,No,Yes,Yes,No,No,filler filler,No\r\n. Product Pricing information:Contact the individual PIP provider to find their prices.. Product purchasing phone number:360-321-1600. \"\"\""

  },

  {

    "role": "user",

    "content": "A Chef needs to cook 9 potatoes. The Chef has already cooked 7.  If each potato takes 3 minutes to cook, how long will it take the Chef to cook the rest?"

  },

  {

    "role": "assistant",

    "content": "It will take 6 minutes to cook the remaining 2 potatoes. \r \r Footnote - Lets think step by step. The Chef has already cooked 7 potatoes. It took 7 times 3 which is 21 minutes total to cook these 7 potatoes. That means it will take 3 more minutes to cook each of the remaining 2 potatoes. 3 times 2 is 6 total minutes."

  },

  {

    "role": "user",

    "content": "Provide me a list of all PIP providers that service South Whidbey Island as well as summarizing the total count.  \r "

  }

]

Response return from ChatCompletions API call:

The PIP providers that service South Whidbey Island are as follows: 1. Azrael R 2. Barbara D 3. Barbara M 4. Bernadete D 5. Briar R 6. Caroline F 7. Cathleen I 8. Charlton W 9. Christine L 10. Cindy H 11. Colleen M 12. Crystel M 13. Darcy L 14. Debbie N 15. Deborah S 16. Debra W 17. Debrax B 18. Dominique K The total count of PIP providers that service South Whidbey Island is 18.

Select Response data values:

Total tokens:1729  Prompt tokens:1590  Completion tokens:139

Duration:10450.704802 milliseconds

Model:gpt-4-1106-preview