Scroll ListView in FutureBuilder before displaying results?

I have a ListView inside a FutureBuilder with an attached ScrollController. How can scroll the ListView to a given position after fetching the data and before displaying it to the user?

Here is my code:

  @override
  Widget build(BuildContext context) {
    return FutureBuilder<List<Conversation>?>(
      future: loadConversations(),
      builder: (BuildContext context, snapshot) {
        if (snapshot.connectionState != ConnectionState.done) {
          return CircularProgressIndicator();
        }
        if (snapshot.hasError) {
          throw snapshot.error!;
        }
        if (!snapshot.hasData) {
          throw Exception('no data found');
        }
        if (snapshot.data!.isEmpty) {
          return Padding(
            padding: const EdgeInsets.all(20.0),
            child: Text('no_conversations'),
          );
        }

        WidgetsBinding.instance!.addPostFrameCallback((_) {
          _scrollController.jumpTo(conversationScrollOffset);
        });

        return ListView.builder(
          itemCount: snapshot.data!.length,
          controller: _scrollController,
          itemBuilder: (context, index) => ConversationTile(
            conversation: snapshot.data![index],
          ),
        );
      },
    );
  }

This approach with addPostFrameCallback works but it is very ugly because the user can see the scrolling / jumping of the ListView. Is there any way to show the user the list with the correct conversationScrollOffset from the beginning e.g. from the first visible frame of the ListView?