Spot the Difference: Your Bucket is Leaking A Novel Methodology to Expose A/B Testing Effortlessly

Abstract

A/B testing - also known as bucket testing - allows an organization to evaluate the performance/impact of new features implemented on their website by exposing a small fraction of visitors to them.

In this paper, we propose a novel methodology that can reveal an ongoing bucket testing and the various features being tested. To evaluate the effectiveness of our proposed methodology, we began with testing the homepages of seven popular websites. We discover that four of them were actively performing bucket testing during our experiments, and we successfully spot different features being tested. Moreover, to investigate the factors that might affect bucket testing, we setup another experiment. Here, we request web pages from different browsers and record several features of server response, e.g., cookies set by the server, IP and port address of the responding server, and response time. We observe variations in the response time for different browsers, which suggest that the type of user agent plays an important role. Finally, we showcase the captured bucket-elements and release our dataset that can serve as ground truth for future investigations in this direction of research.

People

News

Paper/Citation

   Mauro Conti, Ankit Gangwal, Sarada Prasad Gochhayat, Gabriele Tolomei.
   Spot the Difference: Your Bucket is Leaking - A Novel Methodology to Expose A/B Testing Effortlessly.
   In Proceedings of the 4th IEEE Workshop on Security and Privacy in the Cloud
   (IEEE CNS 2018 workshop: SPC 2018), in press, Beijing, China, June 1, 2018.

Dataset

Screenshots

  1. Yahoo: (i) Firefox promotion (Figure 5)   (ii) Page without promotion
  2. Facebook: (i) "WorldMap" image (Figure 6a)   (ii) Multimedia (Figure 6b)
  3. Apple: (i) ipad-1 (Figure 7a)   (ii) ipad-2 (Figure 7b)   (iii) ipad-3 (Figure 7c)   (iv) Accessories-1 (Figure 8a) (v) Accessories-2 (Figure 8b)  
  4. Transferwise: (i) Logo (Figure 9a)   (ii) "Roof" image (Figure 9b)   (iii) "Train" image (Figure 9c)
  5. Booking:
    1. Logo: (i) Logo only   (ii) Logo with text
    2. Text for no. of rooms: (i) "Rooms"   (ii) "Rooms/ units"
    3. Text for "Search" button: (i) "Search"   (ii) "Take me away!"
    4. "Subscribe" element: (i) Light image   (ii) Dark image
    5. Text for sub-menu under "Accommodation" menu item: (i) "Deals"   (ii) "Find deals"   (iii) "Deals for you"
    6. "Referral" element: (i) Image-1   (ii) Image-2   (iii) Image-3
    7. "Feature" element: (i) Compact   (ii) Detailed
    8. "Free cancellation" element: (i) Type-1
    9. "Location" for listings: (i) Type-1
  6. Custom page:
    Version 1 and 3 of the web page are identical while version 2 has a total of five changes compared to version 1. The changes are: Replacement of profile image with an image; replacement of image for QR-code with another image; deletion of text and icons in the social menu; and insertion of text in two dates mentioned in the lower sections of the page.
    1. Version 1 and 3 (1 & 3)
    2. Version 2 (2)
    3. Highlighted differences (12)
    4. No difference (13)
    5. Report for identical versions (No_Diff)
    6. Report for non-identical versions (Tag_Diff)