The new QA in town: chatGPT

chatGPT

You must be living under a rock if you have not yet heard about chatGPT. Amazing things are being told about it everyday, amazing prompts are being posted on Social Media. And as no good deed goes unpunished, scary things like bogus PhD. Thesis, plagiarism at academics are getting reported too.

As usual a section of the (tech) influencers have already envisioned the dooms day for all the developers around the world and if we go by them we will be job less soon. Well this threat is not new, many IDEs and code generator plugins, although not using AI, have given us the scare a decade ago.

Code development for a project or a product is not only about writing code. Modularity, reuse, good architecture and design are few of the many aspects that a developer takes into consideration. Looking at a few chatGPT prompts on social media for auto code or unit test generation I am fairly confident that for those aspects you still need to hire a human developer (although not sure how long).

Nevertheless to satisfy my itchy curiosity I wanted to try out few things on my own. I wanted to see how chatGPT absorbs context over a few continuous back to back chats on a single prompt. Then if I try to get some tangible result out of it, how good is at that.

I decided to use an already existing, very popular code base — Spring Pet Clinic Sample Application. Overview for the sample app can be found here.

Lets use gitpod to download, build and deploy the Spring Pet Clinic code on the fly.

Open gitpod button

Gitpod Cloud Workspace

Run the Spring application

Gitpod provided runtime URL to access the Spring application

Access the Spring Pet Clinic app from local browser

Now that I have the Spring Pet Clinic up and running in less than 5 minutes lets focus on curating the chatGPT prompt.

It knows!

Lets get to the point!

It suggests general steps for writing Selenium test

Hey show me the code!

Generates Add Owner test and asking for more details

Now although it said earlier that it knows about the Spring Petclinic Sample application it is reluctant to do its magic and asks for more details. Nevertheless the Add Owner test case it wrote (below) is fair enough.

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class PetClinicTest {
  private WebDriver driver;

  @Before
  public void setUp() {
    // setup the chrome driver
    System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");
    driver = new ChromeDriver();
  }

  @Test
  public void testAddNewOwner() {
    // navigate to the add new owner page
    driver.get("http://localhost:8080/owners/new");

    // fill in the form
    driver.findElement(By.id("firstName")).sendKeys("John");
    driver.findElement(By.id("lastName")).sendKeys("Doe");
    driver.findElement(By.id("address")).sendKeys("123 Main St");
    driver.findElement(By.id("city")).sendKeys("Anytown");
    driver.findElement(By.id("telephone")).sendKeys("123-456-7890");

    // submit the form
    driver.findElement(By.cssSelector("button[type='submit']")).click();

    // assert that the owner was added successfully
    assertTrue(driver.getPageSource().contains("Owner added successfully"));
  }

  @After
  public void tearDown() {
    driver.quit();
  }
}

Lets give some more context and to the point test cases.

Generate Selenium code for testing the following test cases on Spring Petclinic sample application

1. Test Home page. Open the Spring Petclinic sample application at https://8080-springproje-springpetcl-n727riuuaja.ws-us82.gitpod.io/.
2. Test available menu links on Home page. Validate on the home page following menus are available - Home, Find Owners, Veterinarians, Error.
3. Test Find Owners. From Home Page click on Find Owners menu link. Validate Last Name input box, Find Owner button and Add Owner button is available.
4. Test Add Owner 1. In the Find Owners page, click on Add Owner button. It should open a page with input for First Name, Last Name, Address, City, Telephone. It should also have the Add Owner button.
5. Test Add Owner 2. On the Add Owner page, fill up First Name as Indranil, Last Name as Majumder, Address as India, Telephone as 9999999999. Click on Add Owner button. Validate that on adding the owner the Owner Information is displayed on a table. Also Edit Owner and Add New Pet button is displayed.
6. Test Veterinarians list. From Home Page click Veterinarians menu link. Validate a table of Veterinarians are displayed.

Bingo!

It starts of generating the code…

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import static org.junit.Assert.assertTrue;

public class PetClinicTest {
  private WebDriver driver;

  @Before
  public void setUp() {
    // setup the chrome driver
    System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");
    driver = new ChromeDriver();
  }

  @Test
  public void testHomePage() {
    // navigate to the home page
    driver.get("https://8080-springproje-springpetcl-n727riuuaja.ws-us82.gitpod.io/");

    // assert that the page title is correct
    assertTrue(driver.getTitle().equals("PetClinic :: a Spring Framework demonstration"));
  }

  @Test
  public void testMenuLinks() {
    // navigate to the home page
    driver.get("https://8080-springproje-springpetcl-n727riuuaja.ws-us82.gitpod.io/");

    // assert that the Home, Find Owners, Veterinarians, and Error links are available
    assertTrue(driver.findElement(By.linkText("Home")).isDisplayed());
    assertTrue(driver.findElement(By.linkText("Find Owners")).isDisplayed());
    assertTrue(driver.findElement(By.linkText("Veterinarians")).isDisplayed());
    assertTrue(driver.findElement(By.linkText("Error")).isDisplayed());
  }

  @Test
  public void testFindOwners() {
    // navigate to the home page
    driver.get("https://8080-springproje-springpetcl-n727riuuaja.ws-us82.gitpod.io/");

    // click on the Find Owners link
    driver.findElement(By.linkText("Find Owners")).click();

    // assert that the Last Name input box, Find Owner button, and Add Owner button are available
    assertTrue(driver.findElement(By.id("lastName")).isDisplayed());
    assertTrue(driver.findElement(By.cssSelector("button[type='submit']")).isDisplayed());
    assertTrue(driver.findElement(By.linkText("Add Owner")).isDisplayed());
  }

  @Test
  public void testAddOwner1() {
    // navigate to the find owners page
    driver.get("https://8080-springproje-springpetcl-n727riuuaja.ws-us82.gitpod.io/owners/find");

    // click on the Add Owner button
    driver.findElement(By.linkText("Add Owner")).click();

    // assert that the First Name, Last Name, Address, City, Telephone input boxes and Add Owner button are available
    assertTrue(driver.findElement(By.id("firstName")).isDisplayed());

Notice the code is incomplete, it seems that the OpenAI servers is having too many network timeouts for overuse. Hence it is better to submit the test cases one by one

Add Owner

Good note with the assumptions from chatGPT

Full code….

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import static org.junit.Assert.assertTrue;

public class PetClinicTest {
  private WebDriver driver;

  @Before
  public void setUp() {
    // setup the chrome driver
    System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");
    driver = new ChromeDriver();
  }

  @Test
  public void testHomePage() {
    // navigate to the home page
    driver.get("https://8080-springproje-springpetcl-n727riuuaja.ws-us82.gitpod.io/");

    // assert that the page title is correct
    assertTrue(driver.getTitle().equals("PetClinic :: a Spring Framework demonstration"));
  }

  @Test
  public void testAddOwner1() {
    // navigate to the find owners page
    driver.get("https://8080-springproje-springpetcl-n727riuuaja.ws-us82.gitpod.io/owners/find");

    // click on the Add Owner button
    driver.findElement(By.linkText("Add Owner")).click();

    // assert that the First Name, Last Name, Address, City, Telephone input boxes and Add Owner button are available
    assertTrue(driver.findElement(By.id("firstName")).isDisplayed());
    assertTrue(driver.findElement(By.id("lastName")).isDisplayed());
    assertTrue(driver.findElement(By.id("address")).isDisplayed());
    assertTrue(driver.findElement(By.id("city")).isDisplayed());
    assertTrue(driver.findElement(By.id("telephone")).isDisplayed());
    assertTrue(driver.findElement(By.cssSelector("button[type='submit']")).isDisplayed());
  }

  @After
  public void tearDown() {
    driver.quit();
  }
}

Lets run the auto generated code to test its authenticity…

Head over to Browserstack and follow all the instructions there to test the code.

testng-browserstack/ChatGPTTest.java at local-master · indranil32/testng-browserstack

Note: Considerable amount of work is required in terms of configuration, dependency management and tweaking the code to make the code run. But it eventually runs.

Definitely there is a lot of scope for improvement in terms of how we chat with chatGPT to provide better context setting, be more specific about what we want it to help us with. Like this post. Follow the posts there to know more.

But one thing is clear, its indeed revolutionary and the fact that this can be achieved while chatting in natural spoken language and not some abstruse coding jargon is a big departure from previous similar efforts.

Note: Remember we already have frameworks like Cucumber which executes specifications written in plain text.

For sure the future version of this is going to give all Developers and QAs, worth their salt, a good run for their money.

All hail #chatGPT. All hail #openAI.

Few things I observed :-

1. Right now it is very chatty.
2. It doesn’t like very open ended queries.
3. It asks for specific details, which is a good thing.