Running Krinder on CircleCI

It feels great when your main branch got a new PR merged, a CI picks it up and run the test for you.

In most of the product development cycles, running end to end integration test (that integrates user interface to back-end, machine learning, etc) at the product code build time is going to need a precisely defined test groups and suites. There can be some tens of microservices and multiple front-end components in the system built and deployed to the test environment. Running unit tests and partial integration tests at the build time will alleviate the burden of running full integration tests at the time of one microservice deployment. Test pyramid illustrates that concept.

For example, testing user input length limit, use of emoticon unicode, or testing microservice’s math calculation per given input exploring boundary conditions can be easily covered in the unit tests. End-to-end integration tests should cover more business critical workflow. System can malfunction for many reason even when all unit tests passed. A microservice might have changed the response format or returned empty string where it used to return null (and even fixed the unit tests without telling front-end folks). End to end test coverage should be more of a simulation of what the end users do, and verify the system response, or how stored data through that interaction affects the future system behavior.

So there can be multiple layers of continuous integration, but if an end-to-end test framework like Krinder gets to run tests on CI, it would most likely to be run by a scheduled trigger with combination of other sub group triggers.

• When new Krinder code is pushed – run a smoke test group
• When new front-end code build was deployed to the test target environment – run a front-end test group
• When a microservice build was deployed – run a workflow test group
• Nightly trigger to run all groups

CircleCI

CircleCI provides 6,000 build minutes per month in the free tier and it’s a great platform to run a public repo testing. For a VPN guarded test environment, CircleCI also provides detailed guidelines. The tricky bit was to make CircleCI Insights to pick up the JUnit format test reports generated by TestNG surefire plugin. CircleCI config.yaml file looks like this.

version: 2.1
jobs:
  build-and-test:
    docker:
      - image: markhobson/maven-chrome:latest
    steps:
      - checkout
      - run:
          name: Build
          command: mvn -B -DskipTests clean package
      - run:
          name: Test
          command: mvn test -DbrowserOptions=headless -DchromeDriverPath=/usr/bin/chromedriver
      - run:
          name: Save test results
          command: |
            mkdir -p ~/test-results/junit/
            find . -type f -regex ".*/target/surefire-reports/junitreports/.*xml" -exec cp {} ~/test-results/junit/ \;
          when: always
      - store_artifacts:
          path: ~/test-results
      - store_test_results:
          path: ~/test-results

workflows:
  sample:
    jobs:
      - build-and-test

The job history looks like this in CircleCI.

And the job workflow looks like this.