Teaming up for the first time since 2018, and probably the last time, they lost 7-6 (7-5) 6-4 to Czechs Lucie Hradecka and Linda Noskova.
Together, the American greats have won 14 Grand Slam doubles titles.
Serena Williams, 40, is due to retire after the tournament, while the future plans of elder sister Venus, at 42, remain unknown.
Serena plays Australia's Ajla Tomljanovic in the third round of the women's singles on Friday, but Venus was knocked out in the first round.
Opening the night session on Arthur Ashe Stadium on Thursday, the sisters failed to convert two set points on their opponents' serve at 5-4, with break points having earlier gone begging for both pairs.
Hradecka and Noskova edged the ensuing tie-break on their first set point, with 37-year-old Hradecka wrapping up proceedings with a powerful backhand down the line.
She and Noskova, 20 years her junior, went on to dominate the second set, sealing an early break. They soon held double break point for 4-0 but Noskova couldn't convert from close to the net in a slice of luck for the Williams sisters, who went on to hold and end a five-game losing streak.
They went on to break back but it proved in vain as Hradecka and Noskova broke their serve again on their second match point.
The Williams sisters remain the most recent women's doubles team to hold all four major doubles titles at once with wins at Wimbledon and the US Open in 2009, followed by triumphs at the Australian Open and French Open in 2010.
The last of their Grand Slam doubles titles together came at Wimbledon in 2016, more than 17 years after their first at Roland Garros in 1999, while they also have three Olympic doubles golds.
Venus Williams returned to action for the hard-court swing, appearing in Cincinnati, Toronto and Washington, after nearly a year off following a leg injury, and was awarded a wildcard for the US Open. She is currently ranked 1,504th in the world in the singles, while Serena is 605th.
They have a combined 30 Grand Slam singles titles, 23 of which belong to Serena.