A tightly-fought contest saw the world number 26 earn a 3-6, 6-3, 7-6 (5) win over Sabalenka, who was playing in her first match since her defeat in the Australian Open final last month.
Ons Jabeur also pulled off an impressive victory after beating seventh seed Qinwen Zheng 6-4 6-2 and will meet American Sofia Kenin in the last 16 after she came from behind to defeat compatriot Alycia Parks.
A tough contest saw Gauff finish with 39 unforced errors and only eight winners, and Kostyuk clinched victory after the 2023 US Open champion hit back-to-back double faults while serving for a second-set tiebreak.
Another upset saw Marta Kostyuk book her spot in the last 16 with a straight-sets win over Coco Gauff in Doha.
Alexandrova came from a break down twice in the third set and held serve twice before earning victory in two hours and 36 minutes, setting up a last-16 clash with Elise Mertens.