A bug whose underlying causes are so complex and obscure as to make its behavior appear chaotic or even non-deterministic
Lessons
- realize caching possibility
- look at all data, don't ignore parts
- look for similarities, not only for differences
Verifying
- people from several teams involved
- looking all down the chain till the app
- intermediary API logs
- internal proxy logs
- infosec logs/calls
- debugging proxy calls
- parallel investigation on android side
- interfering POST bug
Second hypothesis
Oracles / data sources
- summarized data from all sources
- occurred on both android and iOS
- not at the same time
- more reasonable explanation:
"GET reply is coming from within the app"
- involving more people from android team
- API logs
- internal proxy logs
- Incapsula/InfoSec logs
- debugging proxy (Charles)
- android studio simulator & logs
- test devices
What is happening?
- phone app receives wrong replies to GET requests
Found!
- caching header was not set
- apps cached GET requests occasionally
- big thanks to all people involved
Reproducibility
First Hypothesis
- non-deterministic
- irregular occurrence
- iOS & Android (no web)
- deciding to focus on android part
- experimenting with reproducibility
- debugging proxy
- not finding GETs in preference API
'Requests are blocked/cached somewhere on the way from app to preference API'
Mandelbug
bug, who doesn't want to be found