Summary
The advice to “browse in incognito mode for cheaper prices” is a classic insider tip of online marketplaces. But how true is this? Would switching to private browsing make a difference in your search results? While browsing privately, less of your personal information (e.g., cookies, browsing history) is available to websites you visit. But does that mean that browsing history actually impacts the price of the goods we search for online? As an example, when two researchers searched from our own profiles, for the same flight itinerary at the same time we see the same seller can offer different prices for the same routing.
How we did it
How can we test how often these differences in prices occur and determine what is the actual dollar impact? One way to answer these questions is by conducting a rigorous study, where different users search for the same goods online, and compare the prices they see. However, one challenge in doing such a study is conducting proper ‘like for like’ comparisons. In the case of investigating flight prices, we compared whether for the exact same flight (day, time, flight number, class, etc.), prices are consistently different across users. To ensure we didn’t see spurious results, we ran our experiments over several months across nearly 112 city pairs (departure and destination) on kayak.com.
In our controlled experiment, we tested how flight prices changed as different users browsed kayak.com; these users were simulated using artificial profiles constructed with different browsing histories. As it turns out, for certain ticket sellers, searching as a different user can have real monetary impact! First, we determined which profile was consistently seeing the best price; this was the incognito profile (i.e., the profile that was simulating searching in incognito mode, or private browsing). Then we looked at, for a specific seller, by how much and how often the other profiles were losing out monetarily to incognito. Comparing different sellers, the results are striking.
What we found
This plot shows, for three (anonymous) sellers, how much more each profile is expected to pay compared to the incognito profile. The dots represent the mean dollar difference where the line length shows the 89% highest posterior density interval in blue. The black dashed line represents the average difference in price between any two profiles. We see that these price impacts can vary across sellers; the seller we anonymize as First Party 1 employs much less differential pricing compared to Third Party 1. Note that we designate sellers as “first party” if the seller of the ticket also operates the flight, and “third party” otherwise. We only show a small subset of sellers here, but not all of them offer such disparate prices to different profiles, suggesting that the price differences are somewhat seller specific. Using our principled technique for detecting price differences, we found that some profiles can be expected to pay multiple dollars more than our control profile. Over time, these differences could lead to large financial losses for consumers. One might also ask if such a pricing system is “fair” to consumers.
Continuing from here
In this work, we conducted an audit that 1) does not require any knowledge about the underlying pricing algorithm, 2) allows us to look at platform versus seller effects, and 3) uses a causal model to understand and compute the effects. Based on this approach, we can more confidently say that it is indeed the difference in browsing behavior which is causing this difference in pricing. For this time period, incognito profiles generally got better prices for flights. This approach could be used for a wide range of goods that typically may not be a subject to traditional disparate impact audits, and could help consumers find cheaper things, possibly by letting people with lower prices buy for those with higher prices (which we will explore in a future post).
To learn more about our data collection process, how we modeled the impact of profiles on prices using our structural causal model, and a similar analysis in the hotel market, check out the paper here and a recording of the talk here. To play with the analysis and scraping code, the GitHub repository here can be used as a starting point.
Resources
@inproceedings{karan2023your,
title={Your Browsing History May Cost You: A Framework for Discovering Differential Pricing in Non-Transparent Markets},
author={Karan, Aditya and Balepur, Naina and Sundaram, Hari},
booktitle={Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency},
pages={717--735},
year={2023}
}